参考外部档案 – http://blog.jialezi.net/?post=153
目的是实现Team(Shared) Drive/Personal Drive到TD的转移,突破每日750G的限制。
原理就是
安装
apt-get install git python3 python3-pip -y
git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && pip3 install -r requirements.txt
如果出现 – ERROR: google-auth 1.16.1 has requirement rsa<4.1,>=3.1.4, but you’ll have rsa 4.1 which is incompatible. 这运行:
pip3 install --upgrade google-auth
生成服务账号Service Accounts – 每个0.75T,最多600个
- 去 https://developers.google.com/drive/api/v3/quickstart/python
- 生成 credentials.json 并拷到 AutoRclone目录下
- scp -P 端口号 /LocalPath/credentials.json root@目标IP:/root/AutoRclone/
再去 https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=342084899348
# 如果希望创建3个新的项目并在新的项目中创建service accounts
python3 gen_sa_accounts.py --quick-setup 3 --new-only
# 创建300个service accounts(3个项目,每个项目100个)
# 将300个service accounts的授权文件下载到accounts文件夹下面
# 如果你想用已有的项目来创建service accounts(不创建新的项目),直接运行
python3 gen_sa_accounts.py --quick-setup 3 -1
# 注意这会覆盖掉已有的service accounts
应该会看到这样的错误
按提示打开,点击”ENABLE“即可。
再次碰上错误
再次打开,点击”ENABLE”
再次运行”python3 gen_sa_accounts.py ……“ 命令 – 终于?了
将Service Accounts加入Google Groups
为了方便管理service accounts,也是为了让我们的Team Drive可以容纳更多的service accounts,我们这里用到了Google Groups。Team Drive最多一次性支持600个SA.
A.1 或者 A.2 这种通过Google Group再加入到Team Drive的方式更推荐,方便管理;B是直接将SA加到Team Drive里面变成内容管理员,会让Drive的管理更加混乱(几百个账号加进去),不推荐。
A.1/A.2/B三者取一就好。
A.1- 对于普通Google账号
创建一个Google Group(https://groups.google.com/)
手动地将service accounts对应的邮箱地址
(可以在json认证文件中找到或在https://console.cloud.google.com/iam-admin/serviceaccounts 每个项目的IAM和管理–服务账号中找到)挨个加进去。但每次只能加10个,每24小时只能加100个。
cd accounts
grep client_email * > ~/service_acounts
vim ~/service_acounts
# 运行 /s/.*\"\(mfc.*com\)\".*/\1/g
或者这个方法
cd # 回到主目录
cat ~/AutoRclone/accounts/*.json | grep "client_email" | awk '{print $2}'| tr -d ',"' | sed 'N;N;N;N;N;N;N;N;N;/^$/d;G' > ~/email.txt
把群组邮箱添加到Team Drive – 在drive.google.com
或A.2 – 对于G Suite管理员 – 还没试,怕把G Suite搞翻车了
按照官方步骤开启Directory API(https://developers.google.com/admin-sdk/directory/v1/quickstart/python),将生成的json文件保存到credentials文件夹下。
在控制面版(https://support.google.com/a/answer/33343?hl=en)里面创建一个群组,创建好你会获得一个类似域名邮箱的地址[email protected]
利用API将service accounts加入Google Groups
python3 add_to_google_group.py -g [email protected]
如果想看参数的具体含义,直接运行python3 add_to_google_group.py -h
将[email protected]添加到Team Drive
B – 直接将SA加入Team Drive(不推荐)
python3 add_to_team_drive.py -d TargetSharedDriveID
root_folder_id_for_shared_omgd
直接使用
参考 – https://github.com/xyou365/AutoRclone#step-5-start-your-task
如若将共享链接(https://drive.google.com/open?id=1nbCm2JydZjnOrzlPYToCIRfRKaHaEQ6X)里的文件保存到自己的 Team Drive 中的 Movie/new2 文件夹中
# 加 -t 作为dry run 试试
python3 rclone_sa_magic.py -s 1nbCm2JydZjnOrzlPYToCIRfRKaHaEQ6X -d 0AIppw6nv_this_is_root_folder_id_for_shared_omgd -dp Movie/new2 -b 1 -e 100
# 将共享盘shared_omgd:test_omgd复制到shared_h1024:test_omgd_dup
python3 rclone_sa_magic.py -s 0AIppw6nv_root_folder_id_for_shared_omgd -sp test_omgd -d 0ANM4a_root_folder_id_for_shared_h1024 -dp test_omgd_dup -b 1 -e 100
python3 rclone_sa_magic.py -h
rclone is detected: /usr/bin/rclone
usage: rclone_sa_magic.py [-h] [-s SOURCE_ID] -d DESTINATION_ID [-sp SOURCE_PATH] [-dp DESTINATION_PATH] [-spi SOURCE_PATH_ID] [-sa SERVICE_ACCOUNT] [-cp] [-p PORT] [-b BEGIN_SA_ID] [-e END_SA_ID] [-c RCLONE_CONFIG_FILE] [-test] [-t]
[--disable_list_r] [--crypt] [--cache]
Copy from source (local/publicly shared drive/Team Drive/) to destination (publicly shared drive/Team Drive).
optional arguments:
-h, --help show this help message and exit
-s SOURCE_ID, --source_id SOURCE_ID
the id of source. Team Drive id or publicly shared folder id
-d DESTINATION_ID, --destination_id DESTINATION_ID
the id of destination. Team Drive id or publicly shared folder id
-sp SOURCE_PATH, --source_path SOURCE_PATH
the folder path of source. In Google Drive or local.
-dp DESTINATION_PATH, --destination_path DESTINATION_PATH
the folder path of destination. In Google Drive.
-spi SOURCE_PATH_ID, --source_path_id SOURCE_PATH_ID
the folder path id (rather than name) of source. In Google Drive.
-sa SERVICE_ACCOUNT, --service_account SERVICE_ACCOUNT
the folder path of json files for service accounts.
-cp, --check_path if check src/dst path or not.
-p PORT, --port PORT the port to run rclone rc. set it to different one if you want to run other instance.
-b BEGIN_SA_ID, --begin_sa_id BEGIN_SA_ID
the begin id of sa for source
-e END_SA_ID, --end_sa_id END_SA_ID
the end id of sa for destination
-c RCLONE_CONFIG_FILE, --rclone_config_file RCLONE_CONFIG_FILE
config file path of rclone
-test, --test_only for test: make rclone print some more information.
-t, --dry_run for test: make rclone dry-run.
--disable_list_r for debug. do not use this.
--crypt for test: crypt remote destination.
--cache for test: cache the remote destination.
通过gclone使用
安装gclone
bash <(wget -qO- https://git.io/gclone.sh)
配置
gclone config和rclone config没什么差别,最重要的就三个 service_account_file, service_account_file_path和 team_drive
[gd_h1024_shared]
type = drive
client_id = 342084899348-可以填申请service_accounts得到的client_id.apps.googleusercontent.com
client_secret = KFxMA申请service_accounts得到的client_secret
scope = drive
service_account_file = /root/AutoRclone/accounts/30a76a14196a41b20086cc62516ca055f2a57d2b.json
service_account_file_path = /root/AutoRclone/accounts/
team_drive = 0ANM4aVovuF_root_folder_id_for_team_drive
# 之前的配置建议保留,加个0,以备所需
[gd_h1024_shared0]
type = drive
client_id = 937512436566-老id-.apps.googleusercontent.com
client_secret = wXoUuLHk1老secret
scope = drive
token = {"access_token":"xxx","token_type":"Bearer","refresh_token":"xxx","expiry":"2020-06-10T14:50:35.167452204-04:00"}
team_drive = 0ANM4aVovuF_root_folder_id_for_team_drive
测试一下
# 这几个都试过了,都可以用
# folder_id => folder_id
gclone copy -vv gd_omgd_shared:{11DRb7-jo5SZsxD0cpVRHxW5ll_uXaPe7} gd_h1024_shared:{11DRb7-jo5SZsxD0cpVRHxW5ll_uXaPe7} --drive-server-side-across-configs
# folder_id => folder_name
gclone copy -vv gd_omgd_shared:{1_vhSkcyfHBxtL6WUxNvBNtBYWX0qWjLh} gd_h1024_shared:/new_folder --drive-server-side-across-configs
# folder_name => folder_name
gclone copy -vv gd_omgd_shared:/test_omgd gd_h1024_shared:/test_omgd_gcloned --drive-server-side-across-configs
比如 https://drive.google.com/drive/folders/1NxfBE9CzJGgeGefoiwrTBPcwGbjfVcnx
gclone copy -vv gd_h1024_shared:{1NxfBE9CzJGgeGefoiwrTBPcwGbjfVcnx} gd_h1024_shared:/333/Must_Have_NewFolderName --drive-server-side-across-configs
文件去重
gclone dedupe newest gd_h1024_shared:{目录的id}