Cannot load the checkpoint

[jmlongriver12]

Describe the bug
It generate the error when running the generate program
To Reproduce
Steps to reproduce the behavior:

1. run "./deepy.py generate.py ./configs/20B.yml -i prompt.txt -o sample_outputs.txt"
2. the error is raised as below:
   Loading extension module utils...
   Loading extension module utils...
   Loading extension module utils...
   Loading extension module utils...
   Traceback (most recent call last):
   File "generate.py", line 91, in
   main()
   File "generate.py", line 33, in main
   model, neox_args = setup_for_inference_or_eval(use_cache=True)
   File "/work/c272987/gpt-neox/megatron/utils.py", line 440, in setup_for_inference_or_eval
   model, _, _ = setup_model_and_optimizer(
   File "/work//gpt-neox/megatron/training.py", line 447, in setup_model_and_optimizer
   neox_args.iteration = load_checkpoint(
   File "/work//gpt-neox/megatron/checkpointing.py", line 239, in load_checkpoint
   checkpoint_name, state_dict = model.load_checkpoint(
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1523, in load_checkpoint
   load_path, client_states = self._load_checkpoint(load_dir,
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1558, in _load_checkpoint
   self.load_module_state_dict(state_dict=checkpoint['module'],
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/engine.py", line 1278, in load_module_state_dict
   self.module.load_state_dir(load_dir=self._curr_ckpt_path, strict=strict)
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/deepspeed/runtime/pipe/module.py", line 571, in load_state_dir
   layer.load_state_dict(torch.load(model_ckpt_path,
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/torch/serialization.py", line 778, in load
   with _open_zipfile_reader(opened_file) as opened_zipfile:
   File "/work//gpt-neox/venv/lib/python3.8/site-packages/torch/serialization.py", line 282, in init
   super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
   RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory
   Traceback (most recent call last):
   File "/opt/conda/lib/python3.8/runpy.py", line 194, in _run_module_as_main
   return _run_code(code, main_globals, None,
   File "/opt/conda/lib/python3.8/runpy.py", line 87, in _run_code
   exec(code, run_globals)
   File "/work/c272987/gpt-neox/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 179, in
   main()
   File "/work/c272987/gpt-neox/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 169, in main
   sigkill_handler(signal.SIGTERM, None) # not coming back
   File "/work/c272987/gpt-neox/venv/lib/python3.8/site-packages/deepspeed/launcher/launch.py", line 147, in sigkill_handler
   raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)

Expected behavior
runs smoothly

Environment (please complete the following information):

• GPUs: 4
• Configs: 20B

[syskn]

RuntimeError: PytorchStreamReader failed reading zip archive: failed finding central directory

When this message shows up, it usually implies that one of the checkpoint files is incomplete (e.g. broken during transfer). Can you check the local files?

[cywjava]

i have error is:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB (GPU 0; 23.70 GiB total capacity; 7.89 GiB already allocated; 39.19 MiB free; 7.89 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[StellaAthena]

i have error is: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB (GPU 0; 23.70 GiB total capacity; 7.89 GiB already allocated; 39.19 MiB free; 7.89 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How big is your GPU? You need a rather large GPU to load a 20B model, and it seems you simply don’t have enough VRAM.

[heukirne]

Hi @StellaAthena , I'm trynig to run inference and finetunning using 20B with 8 x NVIDIA A10G 23GB VRAM and still got the

RuntimeError: CUDA out of memory. Tried to allocate 9.59 GiB (GPU 0; 22.04 GiB total capacity; 14.39 GiB already allocated; 7.00 GiB free; 14.39 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

None of the bellow config work:

"pipe-parallel-size": 8|4|2|1,
"model-parallel-size": 1|2|4|8,

I'm running Version 2.0 of GPT-NeoX
Do you have any tips on how to improve config and be able to run it?

[heukirne]

I was able tu run using HF version
https://github.com/mallorbc/GPTNeoX20B_HuggingFace