自定义数据集(使用HuggingFace)数据集的微调预培训模型

运行环境

python==3.6、transformers==4.6.0、sentencepiece==0.1.94、torch==1.7.0、Flask==1.1.2

C:\Users\Administrator>pip install transformers==4.6.0 Collecting transformers==4.6.0 Downloading https://files.pythonhosted.org/packages/b0/9e/5b80becd952d5f7250eaf8fc64b957077b12ccfe73e9c03d37146ab29712/transformers-4.6.0-py3-none-any.whl (2.3MB) 100% |████████████████████████████████| 2.3MB 12.0MB/s Collecting sacremoses (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/28/78/fef8d089db5b97546fd6d1ff2e813b8544e85670bf3a8c378c9d0250b98d/sacremoses-0.0.53.tar.gz (880kB) 100% |████████████████████████████████| 880kB 16.1MB/s Collecting filelock (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/84/ce/8916d10ef537f3f3b046843255f9799504aa41862bfa87844b9bdc5361cd/filelock-3.4.1-py3-none-any.whl Collecting tqdm>=4.27 (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/47/bb/849011636c4da2e44f1253cd927cfb20ada4374d8b3a4e425416e84900cc/tqdm-4.64.1-py2.py3-none-any.whl (78kB) 100% |████████████████████████████████| 81kB 5.4MB/s Collecting regex!=2019.12.17 (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/12/79/0c4d1d27c042bc6b63dcb61ff3780843ac6696be517ef7416c6d3e545bf4/regex-2022.9.13-cp36-cp36m-win_amd64.whl (279kB) 100% |████████████████████████████████| 286kB 16.6MB/s Collecting requests (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/2d/61/08076519c80041bc0ffa1a8af0cbd3bf3e2b62af10435d269a9d0f40564d/requests-2.27.1-py2.py3-none-any.whl (63kB) 100% |████████████████████████████████| 71kB 4.6MB/s Collecting dataclasses; python_version < "3.7" (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/fe/ca/75fac5856ab5cfa51bbbcefa250182e50441074fdc3f803f6e76451fab43/dataclasses-0.8-py3-none-any.whl Collecting importlib-metadata; python_version < "3.8" (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/a0/a1/b153a0a4caf7a7e3f15c2cd56c7702e2cf3d89b1b359d1f1c5e59d68f4ce/importlib_metadata-4.8.3-py3-none-any.whl Collecting packaging (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/05/8e/8de486cbd03baba4deef4142bd643a3e7bbe954a784dc1bb17142572d127/packaging-21.3-py3-none-any.whl (40kB) 100% |████████████████████████████████| 40kB 2.9MB/s Collecting huggingface-hub==0.0.8 (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl Collecting numpy>=1.17 (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/ea/bc/da526221bc111857c7ef39c3af670bbcf5e69c247b0d22e51986f6d0c5c2/numpy-1.19.5-cp36-cp36m-win_amd64.whl (13.2MB) 100% |████████████████████████████████| 13.2MB 4.6MB/s Collecting tokenizers<0.11,>=0.10.1 (from transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/88/32/ace6b9ebea234ca645812b84118b0756bbc5a7b4efd4a476f7e974a7d914/tokenizers-0.10.3-cp36-cp36m-win_amd64.whl (2.0MB) 100% |████████████████████████████████| 2.0MB 11.4MB/s Collecting six (from sacremoses->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl Collecting click (from sacremoses->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/4a/a8/0b2ced25639fb20cc1c9784de90a8c25f9504a7f18cd8b5397bd61696d7d/click-8.0.4-py3-none-any.whl (97kB) 100% |████████████████████████████████| 102kB 7.7MB/s Collecting joblib (from sacremoses->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/7c/91/d3ba0401e62d7e42816bc7d97b82d19c95c164b3e149a87c0a1c026a735e/joblib-1.1.1-py2.py3-none-any.whl (309kB) 100% |████████████████████████████████| 317kB 20.8MB/s Collecting importlib-resources; python_version < "3.7" (from tqdm>=4.27->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/24/1b/33e489669a94da3ef4562938cd306e8fa915e13939d7b8277cb5569cb405/importlib_resources-5.4.0-py3-none-any.whl Collecting colorama; platform_system == "Windows" (from tqdm>=4.27->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/77/8b/7550e87b2d308a1b711725dfaddc19c695f8c5fa413c640b2be01662f4e6/colorama-0.4.5-py2.py3-none-any.whl Collecting certifi>=2017.4.17 (from requests->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/1d/38/fa96a426e0c0e68aabc68e896584b83ad1eec779265a028e156ce509630e/certifi-2022.9.24-py3-none-any.whl (161kB) 100% |████████████████████████████████| 163kB 27.2MB/s Collecting urllib3<1.27,>=1.21.1 (from requests->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/6f/de/5be2e3eed8426f871b170663333a0f627fc2924cc386cd41be065e7ea870/urllib3-1.26.12-py2.py3-none-any.whl (140kB) 100% |████████████████████████████████| 143kB 22.0MB/s Collecting idna<4,>=2.5; python_version >= "3" (from requests->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl (61kB) 100% |████████████████████████████████| 71kB 10.3MB/s Collecting charset-normalizer~=2.0.0; python_version >= "3" (from requests->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/06/b3/24afc8868eba069a7f03650ac750a778862dc34941a4bebeb58706715726/charset_normalizer-2.0.12-py3-none-any.whl Collecting typing-extensions>=3.6.4; python_version < "3.8" (from importlib-metadata; python_version < "3.8"->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/45/6b/44f7f8f1e110027cf88956b59f2fad776cca7e1704396d043f89effd3a0e/typing_extensions-4.1.1-py3-none-any.whl Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/bd/df/d4a4974a3e3957fd1c1fa3082366d7fff6e428ddb55f074bf64876f8e8ad/zipp-3.6.0-py3-none-any.whl Collecting pyparsing!=3.0.5,>=2.0.2 (from packaging->transformers==4.6.0) Downloading https://files.pythonhosted.org/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl (98kB) 100% |████████████████████████████████| 102kB 8.6MB/s Installing collected packages: regex, six, colorama, typing-extensions, zipp, importlib-metadata, click, joblib, importlib-resources, tqdm, sacremoses, filelock, certifi, urllib3, idna, charset-normalizer, requests, dataclasses, pyparsing, packaging, huggingface-hub, numpy, tokenizers, transformers Running setup.py install for sacremoses ... done Successfully installed certifi-2022.9.24 charset-normalizer-2.0.12 click-8.0.4 colorama-0.4.5 dataclasses-0.8 filelock-3.4.1 huggingface-hub-0.0.8 idna-3.4 importlib-metadata-4.8.3 importlib-resources-5.4.0 joblib-1.1.1 numpy-1.19.5 packaging-21.3 pyparsing-3.0.9 regex-2022.9.13 requests-2.27.1 sacremoses-0.0.53 six-1.16.0 tokenizers-0.10.3 tqdm-4.64.1 transformers-4.6.0 typing-extensions-4.1.1 urllib3-1.26.12 zipp-3.6.0 You are using pip version 18.1, however version 21.3.1 is available. You should consider upgrading via the 'python -m pip install --upgrade pip' command. C:\Users\Administrator>