# Preparation

This project extends Osprey to support multiple pixel-level vision-language tasks.

## Installation

Follow the Osprey installation guide:

```bash
# Clone the repository
git clone https://github.com/CircleRadon/Osprey.git
cd Osprey

# Create conda environment
conda create -n osprey python=3.10 -y
conda activate osprey

# Install packages
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

# Install additional packages for training
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

# Set PYTHONPATH
export PYTHONPATH=`pwd`:$PYTHONPATH
```

## Model Preparation

Download the following pre-trained models to `./checkpoints/`:

- **Qwen2.5-VL-3B-Instruct**: Base vision-language model
- **SAM weights**: `sam_vit_h_4b8939.pth` or `sam_vit_b_01ec64.pth`

```bash
mkdir -p checkpoints
# Download Qwen2.5-VL-3B-Instruct from HuggingFace
# Download SAM weights from https://github.com/facebookresearch/segment-anything
```

---

# Dataset


## 1. DAM (Describe Anything Model) Data

Used for region-level dense captioning tasks, containing multiple sub-datasets:

- **COCOStuff**: COCO stuff segmentation data
- **LVIS**: Large Vocabulary Instance Segmentation
- **Mapillary**: Street-view segmentation data
- **OpenImages**: Open Images dataset
- **PACO**: Parts and Attributes dataset
- **SAM**: Segment Anything data

Data format: Each sub-dataset contains `annotations.json` and `images/` directory.

---

## 2. MUSE Data

Multi-object reasoning segmentation dataset for reasoning segmentation based on natural language descriptions.

| File | Description |
| --- | --- |
| MUSE_train.json | Training set |
| MUSE_val.json | Validation set |
| MUSE_test_less.json | Test set (few objects) |
| MUSE_test_many.json | Test set (many objects) |

Image source: COCO train2014/train2017/val2017

---

## 3. GCG (Grounded Conversation Generation) Data

Used for grounded conversation generation tasks, from the GLaMM project.

| Dataset | Annotation File | Image Path |
| --- | --- | --- |
| GranDf | GranDf_HA_GCG_train.json | glamm_data/images/grandf/train |
| OpenPsg | OpenPsgGCG_train.json | glamm_data/images/coco2017 |
| Flickr30k | flickr_mergedGT_GCG_train.json | glamm_data/images/flickr30k |
| RefCOCOg | RefCOCOg_GCG_train.json | glamm_data/images/coco2014/train2014 |

---

## 4. Pixel2Cap Data

Pixel-level captioning data based on COCO panoptic segmentation.

| File | Description |
| --- | --- |
| pix2cap_coco_train.json | Annotation file |
| coco/train2017 | Image directory |
| coco/panoptic_train2017 | Panoptic segmentation annotations |

---

## 5. RefSeg (Referring Segmentation) Data

Classic referring expression segmentation datasets.

| Dataset | splitBy | Description |
| --- | --- | --- |
| RefCOCO | unc | COCO images, phrase referring |
| RefCOCO+ | unc | No spatial words |
| RefCOCOg | umd | Long sentence descriptions |

Image source: COCO 2014 train2014

---

## 6. Osprey-724K Original Data

724K region-level vision-language data from the Osprey project.

| File | Size |
| --- | ---: |
| osprey_short_form.json | 57 MB |
| osprey_conversation.json | 106 MB |
| osprey_detail_description.json | 63.4 MB |
| osprey_part_level.json | 153 MB |
| osprey_lvis_positive_negative.json | 140 MB |

---

## 7. Other Datasets

- **VG (Visual Genome)**: vg_train_with_mask.json
- **VCR**: Visual Commonsense Reasoning
- **Pascal Part**: pascal_part/train.json
- **PartImageNet**: partImagenet/train_format.json

---

## Data Directory Structure

Organize all data under `./data` directory:

```
data/
├── DAM_data/                    # DAM data
│   ├── COCOStuff/
│   │   ├── annotations.json
│   │   └── images/
│   ├── LVIS/
│   ├── Mapillary/
│   ├── OpenImages/
│   ├── PACO/
│   └── SAM/
├── muse/                        # MUSE data
│   ├── MUSE_train.json
│   ├── MUSE_val.json
│   ├── MUSE_test_less.json
│   └── MUSE_test_many.json
├── glamm_data/                  # GCG data
│   ├── annotations/
│   │   ├── GranDf_HA_GCG_train.json
│   │   ├── OpenPsgGCG_train.json
│   │   ├── flickr_mergedGT_GCG_train.json
│   │   └── RefCOCOg_GCG_train.json
│   └── images/
│       ├── grandf/
│       ├── coco2017/
│       ├── coco2014/
│       └── flickr30k/
├── pix2cap_data/                # Pixel2Cap data
│   ├── annotations/
│   │   └── pix2cap_coco_train.json
│   └── coco/
│       ├── train2017/
│       └── panoptic_train2017/
├── refer_seg/                   # RefSeg data
│   ├── refcoco/
│   ├── refcoco+/
│   ├── refcocog/
│   └── images/
│       └── mscoco/
├── Osprey-724K/                 # Osprey original data
│   ├── osprey_short_form.json
│   ├── osprey_conversation.json
│   ├── osprey_detail_description.json
│   ├── osprey_part_level.json
│   └── osprey_lvis_positive_negative.json
├── coco/                        # COCO images
│   ├── train2014/
│   ├── train2017/
│   └── val2017/
├── vg/                          # Visual Genome
│   ├── vg_train_with_mask.json
│   └── image/
├── vcr/                         # VCR
│   ├── train.jsonl
│   └── vcr1images/
└── part_data/                   # Part data
    ├── pascal_part/
    │   ├── train.json
    │   └── VOCdevkit/
    └── partImagenet/
        ├── train_format.json
        └── train/
```

---

## Download Links

- Osprey-724K: [HuggingFace](https://huggingface.co/datasets/AntGroup-MI/Osprey-724K)
- COCO: [train2017](http://images.cocodataset.org/zips/train2017.zip)
- Pascal Part: [train.json](https://huggingface.co/datasets/sunshine-lwt/Osprey-TrainingData/resolve/main/pascalpart_train.json)
- PartImageNet: [train_format.json](https://huggingface.co/datasets/sunshine-lwt/Osprey-TrainingData/resolve/main/partImagenet_train_format.json)
- RefCOCO: [finetune_refcoco_train_with_mask.json](https://huggingface.co/datasets/sunshine-lwt/Osprey-TrainingData/resolve/main/finetune_refcoco_train_with_mask.json)
- VG: [vg_train_with_mask.json](https://huggingface.co/datasets/sunshine-lwt/Osprey-TrainingData/resolve/main/vg_train_with_mask.json)
- VCR: [download](https://visualcommonsense.com/download/)

---

## Training

The training process consists of two stages. Make sure you have prepared all datasets before training.

### Prerequisites

```bash
# Set PYTHONPATH
export PYTHONPATH=`pwd`:$PYTHONPATH
```

### Stage 1: Multi-task Pre-training

Stage 1 trains the model on multiple vision-language tasks including Osprey-724K, VG, RefSeg, MUSE, Pixel2Cap, and GCG datasets.

**Config file**: `FCLM/configs/fine_grained_task.json`

**Datasets used**:
- Osprey-724K (ShortForm, LVISPosNeg, Conversations, DetailedDescription)
- Visual Genome (VG)
- PACO-LVIS Object Classification
- Reason Segmentation
- Referring Segmentation (RefCOCO, RefCOCO+, RefCOCOg)
- MUSE
- Pixel2Cap
- GCG (GranDf, OpenPsg, Flickr30k, RefCOCOg)

**Run Stage 1**:

```bash
bash scripts/FCLM_stage1.sh
```

### Stage 2: DAM Fine-tuning

Stage 2 fine-tunes the Stage 1 checkpoint on DAM (Describe Anything Model) data for region-level dense captioning.

**Config file**: `FCLM/configs/DAM.json`

**Datasets used**:
- DAM_DATA (COCOStuff, LVIS, Mapillary, OpenImages, PACO, SAM)

**Note**: Due to consistency learning on the same sample for both captioning and grounding, the batch size must be set to 1 in Stage 2.

**Run Stage 2**:

```bash
bash scripts/FCLM_stage2.sh
```

### Custom Training

To customize training, modify the config files:

1. **Add/Remove datasets**: Edit `FCLM/configs/fine_grained_task.json`
2. **Adjust hyperparameters**: Modify the shell scripts in `scripts/`

Example config entry:
```json
{
    "type": "refer_seg",
    "base_image_dir": "./data/refer_seg"
}
```

### Output

Training checkpoints will be saved to (configurable in shell scripts):
- Stage 1: `./output/FCLM_stage1`
- Stage 2: `./output/FCLM_stage2`

---

## Acknowledgements

This project is built upon the following excellent works:

- [Osprey](https://github.com/CircleRadon/Osprey) 
- [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) 
- [GLaMM](https://github.com/mbzuai-oryx/groundingLMM) 
- [DAM (Describe Anything Model)](https://github.com/NVlabs/describe-anything) 
