Rclone高级玩法-利用Gclone+服务账号突破日流量750G限制

参考外部档案 – http://blog.jialezi.net/?post=153

目的是实现Team(Shared) Drive/Personal Drive到TD的转移,突破每日750G的限制。

原理就是

安装

apt-get install git python3 python3-pip -y
git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && pip3 install -r requirements.txt

如果出现 – ERROR: google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you’ll have rsa 4.1 which is incompatible. 这运行:

pip3 install --upgrade google-auth

生成服务账号Service Accounts – 每个0.75T,最多600个

  • 去 https://developers.google.com/drive/api/v3/quickstart/python
  • 生成 credentials.json 并拷到 AutoRclone目录下

再去 https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=342084899348

# 如果希望创建3个新的项目并在新的项目中创建service accounts
python3 gen_sa_accounts.py --quick-setup 3 --new-only
# 创建300个service accounts(3个项目,每个项目100个)
# 将300个service accounts的授权文件下载到accounts文件夹下面

# 如果你想用已有的项目来创建service accounts(不创建新的项目),直接运行
python3 gen_sa_accounts.py --quick-setup 3 -1
# 注意这会覆盖掉已有的service accounts

应该会看到这样的错误

按提示打开,点击”ENABLE“即可。

再次碰上错误

再次打开,点击”ENABLE”

再次运行”python3 gen_sa_accounts.py ……“ 命令 – 终于?了

将Service Accounts加入Google Groups

为了方便管理service accounts,也是为了让我们的Team Drive可以容纳更多的service accounts,我们这里用到了Google Groups。Team Drive最多一次性支持600个SA.

A.1- 对于普通Google账号

创建一个Google Group(https://groups.google.com/)

手动地将service accounts对应的邮箱地址

(可以在json认证文件中找到或在https://console.cloud.google.com/iam-admin/serviceaccounts 每个项目的IAM和管理–服务账号中找到)挨个加进去。但每次只能加10个,每24小时只能加100个。

cd accounts
grep client_email * > ~/service_acounts
vim ~/service_acounts
# 运行 /s/.*\"\(mfc.*com\)\".*/\1/g

或者这个方法

cd # 回到主目录
cat ~/AutoRclone/accounts/*.json | grep "client_email" | awk '{print $2}'| tr -d ',"' | sed 'N;N;N;N;N;N;N;N;N;/^$/d;G' > ~/email.txt

把群组邮箱添加到Team Drive – 在drive.google.com

或A.2 – 对于G Suite管理员 – 还没试,怕把G Suite搞翻车了

按照官方步骤开启Directory API(https://developers.google.com/admin-sdk/directory/v1/quickstart/python),将生成的json文件保存到credentials文件夹下。

在控制面版(https://support.google.com/a/answer/33343?hl=en)里面创建一个群组,创建好你会获得一个类似域名邮箱的地址[email protected]

利用API将service accounts加入Google Groups

python3 add_to_google_group.py -g [email protected]

如果想看参数的具体含义,直接运行python3 add_to_google_group.py -h

[email protected]添加到Team Drive

 python3 add_to_team_drive.py -d TargetSharedDriveID

直接使用

参考 – https://github.com/xyou365/AutoRclone#step-5-start-your-task

如若将共享链接(https://drive.google.com/open?id=1nbCm2JydZjnOrzlPYToCIRfRKaHaEQ6X)里的文件保存到自己的 Team Drive 中的 Movie/new2 文件夹中

# 加 -t 作为dry run 试试
python3 rclone_sa_magic.py -s 1nbCm2JydZjnOrzlPYToCIRfRKaHaEQ6X -d 0AIppw6nv_this_is_root_folder_id_for_shared_omgd -dp Movie/new2 -b 1 -e 100 

# 将共享盘shared_omgd:test_omgd复制到shared_h1024:test_omgd_dup
python3 rclone_sa_magic.py -s 0AIppw6nv_root_folder_id_for_shared_omgd -sp test_omgd -d 0ANM4a_root_folder_id_for_shared_h1024 -dp test_omgd_dup -b 1 -e 100
python3 rclone_sa_magic.py -h
rclone is detected: /usr/bin/rclone
usage: rclone_sa_magic.py [-h] [-s SOURCE_ID] -d DESTINATION_ID [-sp SOURCE_PATH] [-dp DESTINATION_PATH] [-spi SOURCE_PATH_ID] [-sa SERVICE_ACCOUNT] [-cp] [-p PORT] [-b BEGIN_SA_ID] [-e END_SA_ID] [-c RCLONE_CONFIG_FILE] [-test] [-t]
                          [--disable_list_r] [--crypt] [--cache]

Copy from source (local/publicly shared drive/Team Drive/) to destination (publicly shared drive/Team Drive).

optional arguments:
  -h, --help            show this help message and exit
  -s SOURCE_ID, --source_id SOURCE_ID
                        the id of source. Team Drive id or publicly shared folder id
  -d DESTINATION_ID, --destination_id DESTINATION_ID
                        the id of destination. Team Drive id or publicly shared folder id
  -sp SOURCE_PATH, --source_path SOURCE_PATH
                        the folder path of source. In Google Drive or local.
  -dp DESTINATION_PATH, --destination_path DESTINATION_PATH
                        the folder path of destination. In Google Drive.
  -spi SOURCE_PATH_ID, --source_path_id SOURCE_PATH_ID
                        the folder path id (rather than name) of source. In Google Drive.
  -sa SERVICE_ACCOUNT, --service_account SERVICE_ACCOUNT
                        the folder path of json files for service accounts.
  -cp, --check_path     if check src/dst path or not.
  -p PORT, --port PORT  the port to run rclone rc. set it to different one if you want to run other instance.
  -b BEGIN_SA_ID, --begin_sa_id BEGIN_SA_ID
                        the begin id of sa for source
  -e END_SA_ID, --end_sa_id END_SA_ID
                        the end id of sa for destination
  -c RCLONE_CONFIG_FILE, --rclone_config_file RCLONE_CONFIG_FILE
                        config file path of rclone
  -test, --test_only    for test: make rclone print some more information.
  -t, --dry_run         for test: make rclone dry-run.
  --disable_list_r      for debug. do not use this.
  --crypt               for test: crypt remote destination.
  --cache               for test: cache the remote destination.

通过gclone使用

安装gclone

bash <(wget -qO- https://git.io/gclone.sh)

配置

gclone config和rclone config没什么差别,最重要的就三个 service_account_file, service_account_file_path和 team_drive

[gd_h1024_shared]
type = drive
client_id = 342084899348-可以填申请service_accounts得到的client_id.apps.googleusercontent.com
client_secret = KFxMA申请service_accounts得到的client_secret
scope = drive
service_account_file = /root/AutoRclone/accounts/30a76a14196a41b20086cc62516ca055f2a57d2b.json
service_account_file_path = /root/AutoRclone/accounts/
team_drive = 0ANM4aVovuF_root_folder_id_for_team_drive

# 之前的配置建议保留,加个0,以备所需
[gd_h1024_shared0]
type = drive
client_id = 937512436566-老id-.apps.googleusercontent.com
client_secret = wXoUuLHk1老secret
scope = drive
token = {"access_token":"xxx","token_type":"Bearer","refresh_token":"xxx","expiry":"2020-06-10T14:50:35.167452204-04:00"}
team_drive = 0ANM4aVovuF_root_folder_id_for_team_drive

测试一下

team(shared) drive to team(shared) drive

# 这几个都试过了,都可以用

# folder_id => folder_id
gclone copy -vv gd_omgd_shared:{11DRb7-jo5SZsxD0cpVRHxW5ll_uXaPe7} gd_h1024_shared:{11DRb7-jo5SZsxD0cpVRHxW5ll_uXaPe7} --drive-server-side-across-configs

# folder_id => folder_name
gclone copy -vv  gd_omgd_shared:{1_vhSkcyfHBxtL6WUxNvBNtBYWX0qWjLh} gd_h1024_shared:/new_folder --drive-server-side-across-configs

# folder_name => folder_name
gclone copy -vv  gd_omgd_shared:/test_omgd gd_h1024_shared:/test_omgd_gcloned --drive-server-side-across-configs

Shared folder to team(shared) drive

比如 https://drive.google.com/drive/folders/1NxfBE9CzJGgeGefoiwrTBPcwGbjfVcnx

gclone copy -vv gd_h1024_shared:{1NxfBE9CzJGgeGefoiwrTBPcwGbjfVcnx} gd_h1024_shared:/333/Must_Have_NewFolderName --drive-server-side-across-configs

文件去重

gclone dedupe newest gd_h1024_shared:{目录的id}

1 thought on “Rclone高级玩法-利用Gclone+服务账号突破日流量750G限制”

  1. Pingback: gd-utils: 一个非常好用且飞快的telegram bot来做Google Drive转存 – 1024 & 4201

Leave a Reply