沃巴蒂
ff22ed3a8b
本次同步 09-12 至 09-28 的代码变更，主要内容： 新增功能 - 修改密码：后端新增 ChangePasswordRequest DTO 与UserController/UserService 对应接口，前端新增 ChangePasswordDialog 并在 Header 接入入口 - 文件预览组件重构：file-preview 目录拆分出 VideoPreview（支持 blobUrl 直传）、 ImagePreview、AudioPreview，index.tsx 统一调度各类型预览器 - 新增 my_ops 算子模块：markitdown_parser、mineru_pdf_parser、text_quality_evaluator - 新增 tools/minio-file-server MinIO 文件服务 功能增强 - 数据清洗：cleaning_task_service/scheduler/routes 重构， data_juicer_executor 与 wrapper 增强，OperatorTable/LogsTable/TaskDetail 同步调整 - 数据标注：annotation sync.py、project.py 接口与schema 更新 - 数据采集、算子市场、首页与菜单路由相关页面调整 - DataJuicer 算子接入（useOperatorOperations、dj_ops_schema） 结构调整 - deepflow/ 更名为 deep-flow/（419 个文件重命名） - 新增 helm 部署编排、GitHub Actions 之外的自建镜像构建脚本 未纳入本次提交 - scripts-backup/、runtime/python-executor-backup/、components_backup/（备份目录） - *.backup*、__pycache__ 产物 - zh_core_web_sm-3.8.0-py3-none-any.whl（46MB 构建产物） - DataX/（内嵌独立仓库，远程亦未收录其源码）
2839 lines
163 KiB
Plaintext
2839 lines
163 KiB
Plaintext
Total registered operators: 166
|
|
|
|
================================================================================
|
|
【算子名称】: alphanumeric_filter
|
|
【算子类】: AlphanumericFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with an alphabet/numeric ratio within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tokenization type=<class 'bool'>, default=False
|
|
min_ratio type=<class 'float'>, default=0.25
|
|
max_ratio type=<class 'float'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: audio_add_gaussian_noise_mapper
|
|
【算子类】: AudioAddGaussianNoiseMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to add Gaussian noise to audio samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_amplitude type=<class 'float'>, default=0.001
|
|
max_amplitude type=<class 'float'>, default=0.015
|
|
p type=<class 'float'>, default=0.5
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: audio_duration_filter
|
|
【算子类】: AudioDurationFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples whose audio durations are within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_duration type=<class 'int'>, default=0
|
|
max_duration type=<class 'int'>, default=9223372036854775807
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: audio_ffmpeg_wrapped_mapper
|
|
【算子类】: AudioFFmpegWrappedMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Wraps FFmpeg audio filters for processing audio files in a dataset.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
filter_name type=typing.Optional[str], default=None
|
|
filter_kwargs type=typing.Optional[typing.Dict], default=None
|
|
global_args type=typing.Optional[typing.List[str]], default=None
|
|
capture_stderr type=<class 'bool'>, default=True
|
|
overwrite_output type=<class 'bool'>, default=True
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: audio_nmf_snr_filter
|
|
【算子类】: AudioNMFSNRFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples whose audio Signal-to-Noise Ratios (SNRs) are within a specified
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_snr type=<class 'float'>, default=0
|
|
max_snr type=<class 'float'>, default=9223372036854775807
|
|
nmf_iter_num type=typing.Annotated[int, Gt(gt=0)], default=500
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: audio_size_filter
|
|
【算子类】: AudioSizeFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples based on the size of their audio files.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_size type=<class 'str'>, default='0'
|
|
max_size type=<class 'str'>, default='1TB'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: average_line_length_filter
|
|
【算子类】: AverageLineLengthFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with average line length within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_len type=<class 'int'>, default=10
|
|
max_len type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: calibrate_qa_mapper
|
|
【算子类】: CalibrateQAMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Calibrates question-answer pairs based on reference text using an API model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
reference_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: calibrate_query_mapper
|
|
【算子类】: CalibrateQueryMapper
|
|
【算子类型】: CalibrateQAMapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Calibrate query in question-answer pairs based on reference text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
reference_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: calibrate_response_mapper
|
|
【算子类】: CalibrateResponseMapper
|
|
【算子类型】: CalibrateQAMapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Calibrate response in question-answer pairs based on reference text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
reference_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: character_repetition_filter
|
|
【算子类】: CharacterRepetitionFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with character-level n-gram repetition ratio within a specific
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
rep_len type=typing.Annotated[int, Gt(gt=0)], default=10
|
|
min_ratio type=<class 'float'>, default=0.0
|
|
max_ratio type=<class 'float'>, default=0.5
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: chinese_convert_mapper
|
|
【算子类】: ChineseConvertMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to convert Chinese text between Traditional, Simplified, and Japanese Kanji.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
mode type=<class 'str'>, default='s2t'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: clean_copyright_mapper
|
|
【算子类】: CleanCopyrightMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Cleans copyright comments at the beginning of text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: clean_email_mapper
|
|
【算子类】: CleanEmailMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Cleans email addresses from text samples using a regular expression.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
pattern type=typing.Optional[str], default=None
|
|
repl type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: clean_html_mapper
|
|
【算子类】: CleanHtmlMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Cleans HTML code from text samples, converting HTML to plain text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: clean_ip_mapper
|
|
【算子类】: CleanIpMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Cleans IPv4 and IPv6 addresses from text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
pattern type=typing.Optional[str], default=None
|
|
repl type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: clean_links_mapper
|
|
【算子类】: CleanLinksMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to clean links like http/https/ftp in text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
pattern type=typing.Optional[str], default=None
|
|
repl type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: detect_character_attributes_mapper
|
|
【算子类】: DetectCharacterAttributesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Takes an image, a caption, and main character names as input to extract the characters' attributes.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
detect_character_locations_mapper_args type=typing.Optional[typing.Dict], default={}
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: detect_character_locations_mapper
|
|
【算子类】: DetectCharacterLocationsMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Given an image and a list of main character names, extract the bounding boxes for each present character.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
mllm_mapper_args type=typing.Optional[typing.Dict], default={}
|
|
image_text_matching_filter_args type=typing.Optional[typing.Dict], default={}
|
|
yoloe_path type=<class 'inspect._empty'>, default='yoloe-11l-seg.pt'
|
|
iou_threshold type=<class 'inspect._empty'>, default=0.7
|
|
matching_score_threshold type=<class 'inspect._empty'>, default=0.4
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: detect_main_character_mapper
|
|
【算子类】: DetectMainCharacterMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Extract all main character names based on the given image and its caption.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
mllm_mapper_args type=typing.Optional[typing.Dict], default={}
|
|
filter_min_character_num type=<class 'int'>, default=0
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: dialog_intent_detection_mapper
|
|
【算子类】: DialogIntentDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Generates user's intent labels in a dialog by analyzing the history, query, and
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
intent_candidates type=typing.Optional[typing.List[str]], default=None
|
|
max_round type=typing.Annotated[int, Ge(ge=0)], default=10
|
|
labels_key type=<class 'str'>, default='dialog_intent_labels'
|
|
analysis_key type=<class 'str'>, default='dialog_intent_labels_analysis'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
candidate_template type=typing.Optional[str], default=None
|
|
analysis_template type=typing.Optional[str], default=None
|
|
labels_template type=typing.Optional[str], default=None
|
|
analysis_pattern type=typing.Optional[str], default=None
|
|
labels_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: dialog_sentiment_detection_mapper
|
|
【算子类】: DialogSentimentDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Generates sentiment labels and analysis for user queries in a dialog.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
sentiment_candidates type=typing.Optional[typing.List[str]], default=None
|
|
max_round type=typing.Annotated[int, Ge(ge=0)], default=10
|
|
labels_key type=<class 'str'>, default='dialog_sentiment_labels'
|
|
analysis_key type=<class 'str'>, default='dialog_sentiment_labels_analysis'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
candidate_template type=typing.Optional[str], default=None
|
|
analysis_template type=typing.Optional[str], default=None
|
|
labels_template type=typing.Optional[str], default=None
|
|
analysis_pattern type=typing.Optional[str], default=None
|
|
labels_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: dialog_sentiment_intensity_mapper
|
|
【算子类】: DialogSentimentIntensityMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to predict user's sentiment intensity in a dialog, ranging from -5 to 5.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
max_round type=typing.Annotated[int, Ge(ge=0)], default=10
|
|
intensities_key type=<class 'str'>, default='dialog_sentiment_intensity'
|
|
analysis_key type=<class 'str'>, default='dialog_sentiment_intensity_analysis'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
analysis_template type=typing.Optional[str], default=None
|
|
intensity_template type=typing.Optional[str], default=None
|
|
analysis_pattern type=typing.Optional[str], default=None
|
|
intensity_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: dialog_topic_detection_mapper
|
|
【算子类】: DialogTopicDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Generates user's topic labels and analysis in a dialog.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
topic_candidates type=typing.Optional[typing.List[str]], default=None
|
|
max_round type=typing.Annotated[int, Ge(ge=0)], default=10
|
|
labels_key type=<class 'str'>, default='dialog_topic_labels'
|
|
analysis_key type=<class 'str'>, default='dialog_topic_labels_analysis'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
candidate_template type=typing.Optional[str], default=None
|
|
analysis_template type=typing.Optional[str], default=None
|
|
labels_template type=typing.Optional[str], default=None
|
|
analysis_pattern type=typing.Optional[str], default=None
|
|
labels_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: document_deduplicator
|
|
【算子类】: DocumentDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using exact matching.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lowercase type=<class 'bool'>, default=False
|
|
ignore_non_character type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: document_minhash_deduplicator
|
|
【算子类】: DocumentMinhashDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using MinHash LSH.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tokenization type=<class 'str'>, default='space'
|
|
window_size type=typing.Annotated[int, Gt(gt=0)], default=5
|
|
lowercase type=<class 'bool'>, default=True
|
|
ignore_pattern type=typing.Optional[str], default=None
|
|
num_permutations type=typing.Annotated[int, Gt(gt=0)], default=256
|
|
jaccard_threshold type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.7
|
|
num_bands type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
num_rows_per_band type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
tokenizer_model type=typing.Optional[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: document_simhash_deduplicator
|
|
【算子类】: DocumentSimhashDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using SimHash.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tokenization type=<class 'str'>, default='space'
|
|
window_size type=typing.Annotated[int, Gt(gt=0)], default=6
|
|
lowercase type=<class 'bool'>, default=True
|
|
ignore_pattern type=typing.Optional[str], default=None
|
|
num_blocks type=typing.Annotated[int, Gt(gt=0)], default=6
|
|
hamming_distance type=typing.Annotated[int, Gt(gt=0)], default=4
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: download_file_mapper
|
|
【算子类】: DownloadFileMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to download URL files to local files or load them into memory.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
download_field type=<class 'str'>, default=None
|
|
save_dir type=<class 'str'>, default=None
|
|
save_field type=<class 'str'>, default=None
|
|
resume_download type=<class 'bool'>, default=False
|
|
timeout type=<class 'int'>, default=30
|
|
max_concurrent type=<class 'int'>, default=10
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: entity_attribute_aggregator
|
|
【算子类】: EntityAttributeAggregator
|
|
【算子类型】: Aggregator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Summarizes a given attribute of an entity from a set of documents.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
entity type=<class 'str'>, default=None
|
|
attribute type=<class 'str'>, default=None
|
|
input_key type=<class 'str'>, default='event_description'
|
|
output_key type=<class 'str'>, default='entity_attribute'
|
|
word_limit type=typing.Annotated[int, Gt(gt=0)], default=100
|
|
max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt_template type=typing.Optional[str], default=None
|
|
example_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: expand_macro_mapper
|
|
【算子类】: ExpandMacroMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Expands macro definitions in the document body of LaTeX samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_entity_attribute_mapper
|
|
【算子类】: ExtractEntityAttributeMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts attributes for given entities from the text and stores them in the sample's
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
query_entities type=typing.List[str], default=[]
|
|
query_attributes type=typing.List[str], default=[]
|
|
entity_key type=<class 'str'>, default='main_entities'
|
|
attribute_key type=<class 'str'>, default='attributes'
|
|
attribute_desc_key type=<class 'str'>, default='attribute_descriptions'
|
|
support_text_key type=<class 'str'>, default='attribute_support_texts'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt_template type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
attr_pattern_template type=typing.Optional[str], default=None
|
|
demo_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_entity_relation_mapper
|
|
【算子类】: ExtractEntityRelationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts entities and relations from text to build a knowledge graph.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
entity_types type=typing.List[str], default=None
|
|
entity_key type=<class 'str'>, default='entity'
|
|
relation_key type=<class 'str'>, default='relation'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
prompt_template type=typing.Optional[str], default=None
|
|
tuple_delimiter type=typing.Optional[str], default=None
|
|
record_delimiter type=typing.Optional[str], default=None
|
|
completion_delimiter type=typing.Optional[str], default=None
|
|
max_gleaning type=typing.Annotated[int, Ge(ge=0)], default=1
|
|
continue_prompt type=typing.Optional[str], default=None
|
|
if_loop_prompt type=typing.Optional[str], default=None
|
|
entity_pattern type=typing.Optional[str], default=None
|
|
relation_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_event_mapper
|
|
【算子类】: ExtractEventMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Extracts events and relevant characters from the text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
event_desc_key type=<class 'str'>, default='event_description'
|
|
relevant_char_key type=<class 'str'>, default='relevant_characters'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_keyword_mapper
|
|
【算子类】: ExtractKeywordMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Generate keywords for the text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
keyword_key type=<class 'str'>, default='keyword'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
prompt_template type=typing.Optional[str], default=None
|
|
completion_delimiter type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_nickname_mapper
|
|
【算子类】: ExtractNicknameMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts nickname relationships in the text using a language model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
nickname_key type=<class 'str'>, default='nickname'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_support_text_mapper
|
|
【算子类】: ExtractSupportTextMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts a supporting sub-text from the original text based on a given summary.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
summary_key type=<class 'str'>, default='event_description'
|
|
support_text_key type=<class 'str'>, default='support_text'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: extract_tables_from_html_mapper
|
|
【算子类】: ExtractTablesFromHtmlMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts tables from HTML content and stores them in a specified field.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tables_field_name type=<class 'str'>, default='html_tables'
|
|
retain_html_tags type=<class 'bool'>, default=False
|
|
include_header type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: fix_unicode_mapper
|
|
【算子类】: FixUnicodeMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Fixes unicode errors in text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
normalization type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: flagged_words_filter
|
|
【算子类】: FlaggedWordFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with flagged-word ratio in a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
tokenization type=<class 'bool'>, default=False
|
|
min_ratio type=<class 'float'>, default=0.0
|
|
max_ratio type=<class 'float'>, default=0.045
|
|
flagged_words_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
use_words_aug type=<class 'bool'>, default=False
|
|
words_aug_group_sizes type=typing.List[typing.Annotated[int, Gt(gt=0)]], default=[2]
|
|
words_aug_join_char type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: frequency_specified_field_selector
|
|
【算子类】: FrequencySpecifiedFieldSelector
|
|
【算子类型】: Selector
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Selector to filter samples based on the frequency of a specified field.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
top_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None
|
|
topk type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
reverse type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: general_field_filter
|
|
【算子类】: GeneralFieldFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples based on a general field filter condition.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
filter_condition type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: general_fused_op
|
|
【算子类】: GeneralFusedOP
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: An explicitly fused operator designed to execute multiple sequential operations (OPs) on
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
batch_size type=<class 'int'>, default=1
|
|
fused_op_list type=typing.Optional[typing.List], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: generate_qa_from_examples_mapper
|
|
【算子类】: GenerateQAFromExamplesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates question and answer pairs from examples using a Hugging Face model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='Qwen/Qwen2.5-7B-Instruct'
|
|
seed_file type=<class 'str'>, default=''
|
|
example_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
similarity_threshold type=<class 'float'>, default=0.7
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
example_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: generate_qa_from_text_mapper
|
|
【算子类】: GenerateQAFromTextMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Generates question and answer pairs from text using a specified model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='alibaba-pai/pai-qwen1_5-7b-doc2qa'
|
|
max_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: human_preference_annotation_mapper
|
|
【算子类】: HumanPreferenceAnnotationMapper
|
|
【算子类型】: LabelStudioAnnotationMapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Operator for human preference annotation using Label Studio.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
label_config_file type=<class 'str'>, default=None
|
|
answer1_key type=<class 'str'>, default='answer1'
|
|
answer2_key type=<class 'str'>, default='answer2'
|
|
prompt_key type=<class 'str'>, default='prompt'
|
|
chosen_key type=<class 'str'>, default='chosen'
|
|
rejected_key type=<class 'str'>, default='rejected'
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_aesthetics_filter
|
|
【算子类】: ImageAestheticsFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with aesthetics scores within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_scorer_model type=<class 'str'>, default=''
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.5
|
|
max_score type=<class 'float'>, default=1.0
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_aspect_ratio_filter
|
|
【算子类】: ImageAspectRatioFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with image aspect ratio within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_ratio type=<class 'float'>, default=0.333
|
|
max_ratio type=<class 'float'>, default=3.0
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_blur_mapper
|
|
【算子类】: ImageBlurMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Blurs images in the dataset with a specified probability and blur type.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
p type=<class 'float'>, default=0.2
|
|
blur_type type=<class 'str'>, default='gaussian'
|
|
radius type=<class 'float'>, default=2
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_captioning_from_gpt4v_mapper
|
|
【算子类】: ImageCaptioningFromGPT4VMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Generates text captions for images using the GPT-4 Vision model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
mode type=<class 'str'>, default='description'
|
|
api_key type=<class 'str'>, default=''
|
|
max_token type=<class 'int'>, default=500
|
|
temperature type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=1.0
|
|
system_prompt type=<class 'str'>, default=''
|
|
user_prompt type=<class 'str'>, default=''
|
|
user_prompt_key type=typing.Optional[str], default=None
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_captioning_mapper
|
|
【算子类】: ImageCaptioningMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Generates image captions using a Hugging Face model and appends them to samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_img2seq type=<class 'str'>, default='Salesforce/blip2-opt-2.7b'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
caption_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_candidate_mode type=<class 'str'>, default='random_any'
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
prompt type=typing.Optional[str], default=None
|
|
prompt_key type=typing.Optional[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_deduplicator
|
|
【算子类】: ImageDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level by exact matching of images.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
method type=<class 'str'>, default='phash'
|
|
consider_text type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_detection_yolo_mapper
|
|
【算子类】: ImageDetectionYoloMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Perform object detection using YOLO on images and return bounding boxes and class
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
imgsz type=<class 'inspect._empty'>, default=640
|
|
conf type=<class 'inspect._empty'>, default=0.05
|
|
iou type=<class 'inspect._empty'>, default=0.5
|
|
model_path type=<class 'inspect._empty'>, default='yolo11n.pt'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_diffusion_mapper
|
|
【算子类】: ImageDiffusionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Generate images using a diffusion model based on provided captions.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_diffusion type=<class 'str'>, default='CompVis/stable-diffusion-v1-4'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
torch_dtype type=<class 'str'>, default='fp32'
|
|
revision type=<class 'str'>, default='main'
|
|
strength type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.8
|
|
guidance_scale type=<class 'float'>, default=7.5
|
|
aug_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
caption_key type=typing.Optional[str], default=None
|
|
hf_img2seq type=<class 'str'>, default='Salesforce/blip2-opt-2.7b'
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_face_blur_mapper
|
|
【算子类】: ImageFaceBlurMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to blur faces detected in images.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
cv_classifier type=<class 'str'>, default=''
|
|
blur_type type=<class 'str'>, default='gaussian'
|
|
radius type=typing.Annotated[float, Ge(ge=0)], default=2
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_face_count_filter
|
|
【算子类】: ImageFaceCountFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with the number of faces within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
cv_classifier type=<class 'str'>, default=''
|
|
min_face_count type=<class 'int'>, default=1
|
|
max_face_count type=<class 'int'>, default=1
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_face_ratio_filter
|
|
【算子类】: ImageFaceRatioFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with face area ratios within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
cv_classifier type=<class 'str'>, default=''
|
|
min_ratio type=<class 'float'>, default=0.0
|
|
max_ratio type=<class 'float'>, default=0.4
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_nsfw_filter
|
|
【算子类】: ImageNSFWFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples whose images have nsfw scores in a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_nsfw_model type=<class 'str'>, default='Falconsai/nsfw_image_detection'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.0
|
|
max_score type=<class 'float'>, default=0.5
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_pair_similarity_filter
|
|
【算子类】: ImagePairSimilarityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep image pairs with similarities between images within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_clip type=<class 'inspect._empty'>, default='openai/clip-vit-base-patch32'
|
|
trust_remote_code type=<class 'inspect._empty'>, default=False
|
|
min_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=0.1
|
|
max_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=1.0
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_remove_background_mapper
|
|
【算子类】: ImageRemoveBackgroundMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to remove the background of images.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
alpha_matting type=<class 'bool'>, default=False
|
|
alpha_matting_foreground_threshold type=<class 'int'>, default=240
|
|
alpha_matting_background_threshold type=<class 'int'>, default=10
|
|
alpha_matting_erode_size type=<class 'int'>, default=10
|
|
bgcolor type=typing.Optional[typing.Tuple[int, int, int, int]], default=None
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_segment_mapper
|
|
【算子类】: ImageSegmentMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Perform segment-anything on images and return the bounding boxes.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
imgsz type=<class 'inspect._empty'>, default=1024
|
|
conf type=<class 'inspect._empty'>, default=0.05
|
|
iou type=<class 'inspect._empty'>, default=0.5
|
|
model_path type=<class 'inspect._empty'>, default='FastSAM-x.pt'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_shape_filter
|
|
【算子类】: ImageShapeFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with image shape (width, height) within specific ranges.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_width type=<class 'int'>, default=1
|
|
max_width type=<class 'int'>, default=9223372036854775807
|
|
min_height type=<class 'int'>, default=1
|
|
max_height type=<class 'int'>, default=9223372036854775807
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_size_filter
|
|
【算子类】: ImageSizeFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Keep data samples whose image size (in Bytes/KB/MB/...) is within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_size type=<class 'str'>, default='0'
|
|
max_size type=<class 'str'>, default='1TB'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_tagging_mapper
|
|
【算子类】: ImageTaggingMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates image tags for each image in the sample.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tag_field_name type=<class 'str'>, default='image_tags'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_text_matching_filter
|
|
【算子类】: ImageTextMatchingFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with image-text matching scores within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_blip type=<class 'str'>, default='Salesforce/blip-itm-base-coco'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.003
|
|
max_score type=<class 'float'>, default=1.0
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_text_similarity_filter
|
|
【算子类】: ImageTextSimilarityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with image-text similarity within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_clip type=<class 'str'>, default='openai/clip-vit-base-patch32'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.1
|
|
max_score type=<class 'float'>, default=1.0
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: image_watermark_filter
|
|
【算子类】: ImageWatermarkFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples whose images have no watermark with high probability.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_watermark_model type=<class 'str'>, default='amrul-hzz/watermark_detector'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
prob_threshold type=<class 'float'>, default=0.8
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: imgdiff_difference_area_generator_mapper
|
|
【算子类】: Difference_Area_Generator_Mapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates and filters bounding boxes for image pairs based on similarity, segmentation,
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
image_pair_similarity_filter_args type=typing.Optional[typing.Dict], default={}
|
|
image_segment_mapper_args type=typing.Optional[typing.Dict], default={}
|
|
image_text_matching_filter_args type=typing.Optional[typing.Dict], default={}
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: imgdiff_difference_caption_generator_mapper
|
|
【算子类】: Difference_Caption_Generator_Mapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates difference captions for bounding box regions in two images.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
mllm_mapper_args type=typing.Optional[typing.Dict], default={}
|
|
image_text_matching_filter_args type=typing.Optional[typing.Dict], default={}
|
|
text_pair_similarity_filter_args type=typing.Optional[typing.Dict], default={}
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: in_context_influence_filter
|
|
【算子类】: InContextInfluenceFilter
|
|
【算子类型】: LLMPerplexityFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep texts based on their in-context influence on a validation set.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None
|
|
task_desc type=<class 'str'>, default=None
|
|
valid_as_demo type=<class 'bool'>, default=False
|
|
n_shot type=typing.Optional[int], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: instruction_following_difficulty_filter
|
|
【算子类】: InstructionFollowingDifficultyFilter
|
|
【算子类型】: LLMPerplexityFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep texts based on their instruction following difficulty (IFD,
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='Qwen/Qwen2.5-0.5B'
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
min_score type=<class 'float'>, default=1.0
|
|
max_score type=<class 'float'>, default=100.0
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: key_value_grouper
|
|
【算子类】: KeyValueGrouper
|
|
【算子类型】: Grouper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Groups samples into batches based on values in specified keys.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
group_by_keys type=typing.Optional[typing.List[str]], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: language_id_score_filter
|
|
【算子类】: LanguageIDScoreFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples in a specific language with a confidence score above a threshold.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=typing.Union[str, typing.List[str]], default=''
|
|
min_score type=<class 'float'>, default=0.8
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: llm_analysis_filter
|
|
【算子类】: LLMAnalysisFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Base filter class for leveraging LLMs to analyze and filter data samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='gpt-4o'
|
|
min_score type=<class 'float'>, default=0.5
|
|
max_score type=<class 'float'>, default=1.0
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
input_keys type=typing.List[str], default=['text']
|
|
field_names type=typing.List[str], default=['Text']
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
field_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
dim_required_keys type=typing.Optional[typing.List[str]], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: llm_difficulty_score_filter
|
|
【算子类】: LLMDifficultyScoreFilter
|
|
【算子类型】: LLMAnalysisFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with high difficulty scores estimated by an LLM.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='gpt-4o'
|
|
min_score type=<class 'float'>, default=0.5
|
|
max_score type=<class 'float'>, default=1.0
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
input_keys type=typing.List[str], default=['text']
|
|
field_names type=typing.List[str], default=['Text']
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
field_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
dim_required_keys type=typing.Optional[typing.List[str]], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: llm_perplexity_filter
|
|
【算子类】: LLMPerplexityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with perplexity scores within a specified range, computed using a
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='Qwen/Qwen2.5-0.5B'
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
min_score type=<class 'float'>, default=1.0
|
|
max_score type=<class 'float'>, default=100.0
|
|
query_template type=typing.Optional[str], default=None
|
|
response_template type=typing.Optional[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: llm_quality_score_filter
|
|
【算子类】: LLMQualityScoreFilter
|
|
【算子类型】: LLMAnalysisFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with a high quality score estimated by a language model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='gpt-4o'
|
|
min_score type=<class 'float'>, default=0.5
|
|
max_score type=<class 'float'>, default=1.0
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
input_keys type=typing.List[str], default=['text']
|
|
field_names type=typing.List[str], default=['Text']
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
field_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
dim_required_keys type=typing.Optional[typing.List[str]], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: llm_task_relevance_filter
|
|
【算子类】: LLMTaskRelevanceFilter
|
|
【算子类型】: LLMAnalysisFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with high relevance scores to validation tasks estimated by an
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='gpt-4o'
|
|
min_score type=<class 'float'>, default=0.5
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None
|
|
task_desc type=typing.Optional[str], default=None
|
|
n_shot type=typing.Optional[int], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: maximum_line_length_filter
|
|
【算子类】: MaximumLineLengthFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with a maximum line length within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_len type=<class 'int'>, default=10
|
|
max_len type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: meta_tags_aggregator
|
|
【算子类】: MetaTagsAggregator
|
|
【算子类型】: Aggregator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Merge similar meta tags into a single, unified tag.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
meta_tag_key type=<class 'str'>, default='dialog_sentiment_labels'
|
|
target_tags type=typing.Optional[typing.List[str]], default=None
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
target_tag_template type=typing.Optional[str], default=None
|
|
tag_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: mllm_mapper
|
|
【算子类】: MllmMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to use MLLMs for visual question answering tasks. This operator uses a Hugging
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='llava-hf/llava-v1.6-vicuna-7b-hf'
|
|
max_new_tokens type=<class 'inspect._empty'>, default=256
|
|
temperature type=<class 'inspect._empty'>, default=0.2
|
|
top_p type=<class 'inspect._empty'>, default=None
|
|
num_beams type=<class 'inspect._empty'>, default=1
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: most_relevant_entities_aggregator
|
|
【算子类】: MostRelevantEntitiesAggregator
|
|
【算子类型】: Aggregator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Extracts and ranks entities closely related to a given entity from provided texts.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
entity type=<class 'str'>, default=None
|
|
query_entity_type type=<class 'str'>, default=None
|
|
input_key type=<class 'str'>, default='event_description'
|
|
output_key type=<class 'str'>, default='most_relevant_entities'
|
|
max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt_template type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: naive_grouper
|
|
【算子类】: NaiveGrouper
|
|
【算子类型】: Grouper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Group all samples in a dataset into a single batched sample.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: naive_reverse_grouper
|
|
【算子类】: NaiveReverseGrouper
|
|
【算子类型】: Grouper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Split batched samples into individual samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
batch_meta_export_path type=<class 'inspect._empty'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: nested_aggregator
|
|
【算子类】: NestedAggregator
|
|
【算子类型】: Aggregator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Aggregates nested content from multiple samples into a single summary.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
input_key type=<class 'str'>, default='event_description'
|
|
output_key type=<class 'str'>, default=None
|
|
max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
sub_doc_template type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: nlpaug_en_mapper
|
|
【算子类】: NlpaugEnMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Augments English text samples using various methods from the nlpaug library.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
sequential type=<class 'bool'>, default=False
|
|
aug_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
delete_random_word type=<class 'bool'>, default=False
|
|
swap_random_word type=<class 'bool'>, default=False
|
|
spelling_error_word type=<class 'bool'>, default=False
|
|
split_random_word type=<class 'bool'>, default=False
|
|
keyboard_error_char type=<class 'bool'>, default=False
|
|
ocr_error_char type=<class 'bool'>, default=False
|
|
delete_random_char type=<class 'bool'>, default=False
|
|
swap_random_char type=<class 'bool'>, default=False
|
|
insert_random_char type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: nlpcda_zh_mapper
|
|
【算子类】: NlpcdaZhMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Augments Chinese text samples using the nlpcda library.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
sequential type=<class 'bool'>, default=False
|
|
aug_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
replace_similar_word type=<class 'bool'>, default=False
|
|
replace_homophone_char type=<class 'bool'>, default=False
|
|
delete_random_char type=<class 'bool'>, default=False
|
|
swap_random_char type=<class 'bool'>, default=False
|
|
replace_equivalent_num type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: optimize_prompt_mapper
|
|
【算子类】: OptimizePromptMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Optimize prompts based on existing ones in the same batch.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='Qwen/Qwen2.5-7B-Instruct'
|
|
gen_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
max_example_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
retry_num type=<class 'int'>, default=3
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
example_template type=typing.Optional[str], default=None
|
|
prompt_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: optimize_qa_mapper
|
|
【算子类】: OptimizeQAMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to optimize question-answer pairs.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='Qwen/Qwen2.5-7B-Instruct'
|
|
is_hf_model type=<class 'bool'>, default=True
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: optimize_query_mapper
|
|
【算子类】: OptimizeQueryMapper
|
|
【算子类型】: OptimizeQAMapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Optimize queries in question-answer pairs to make them more specific and detailed.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='Qwen/Qwen2.5-7B-Instruct'
|
|
is_hf_model type=<class 'bool'>, default=True
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: optimize_response_mapper
|
|
【算子类】: OptimizeResponseMapper
|
|
【算子类型】: OptimizeQAMapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Optimize response in question-answer pairs to be more detailed and specific.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='Qwen/Qwen2.5-7B-Instruct'
|
|
is_hf_model type=<class 'bool'>, default=True
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
qa_pair_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
enable_vllm type=<class 'bool'>, default=False
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
sampling_params type=typing.Optional[typing.Dict], default=None
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: pair_preference_mapper
|
|
【算子类】: PairPreferenceMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to construct paired preference samples by generating a rejected response and its
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern type=typing.Optional[str], default=None
|
|
rejected_key type=<class 'str'>, default='rejected_response'
|
|
reason_key type=<class 'str'>, default='reason'
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: perplexity_filter
|
|
【算子类】: PerplexityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with perplexity score in a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
min_ppl type=<class 'float'>, default=0
|
|
max_ppl type=<class 'float'>, default=1500
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: phrase_grounding_recall_filter
|
|
【算子类】: PhraseGroundingRecallFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples based on the phrase grounding recall of phrases extracted from
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_owlvit type=<class 'str'>, default='google/owlvit-base-patch32'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_recall type=<class 'float'>, default=0.1
|
|
max_recall type=<class 'float'>, default=1.0
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
iou_thr type=<class 'float'>, default=0.5
|
|
large_area_ratio_thr type=<class 'float'>, default=0.95
|
|
conf_thr type=<class 'float'>, default=0.0
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: punctuation_normalization_mapper
|
|
【算子类】: PunctuationNormalizationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Normalizes unicode punctuations to their English equivalents in text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: python_file_mapper
|
|
【算子类】: PythonFileMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Executes a Python function defined in a file on input data.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
file_path type=<class 'str'>, default=''
|
|
function_name type=<class 'str'>, default='process_single'
|
|
batched type=<class 'bool'>, default=False
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: python_lambda_mapper
|
|
【算子类】: PythonLambdaMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper for applying a Python lambda function to data samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lambda_str type=<class 'str'>, default=''
|
|
batched type=<class 'bool'>, default=False
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: query_intent_detection_mapper
|
|
【算子类】: QueryIntentDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Predicts the user's intent label and corresponding score for a given query. The operator
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='bespin-global/klue-roberta-small-3i4k-intent-classification'
|
|
zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en'
|
|
model_params type=typing.Dict, default={}
|
|
zh_to_en_model_params type=typing.Dict, default={}
|
|
label_key type=<class 'str'>, default='query_intent_label'
|
|
score_key type=<class 'str'>, default='query_intent_label_score'
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: query_sentiment_detection_mapper
|
|
【算子类】: QuerySentimentDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Predicts user's sentiment label ('negative', 'neutral', 'positive') in a query.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis'
|
|
zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en'
|
|
model_params type=typing.Dict, default={}
|
|
zh_to_en_model_params type=typing.Dict, default={}
|
|
label_key type=<class 'str'>, default='query_sentiment_label'
|
|
score_key type=<class 'str'>, default='query_sentiment_label_score'
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: query_topic_detection_mapper
|
|
【算子类】: QueryTopicDetectionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Predicts the topic label and its corresponding score for a given query. The input is
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='dstefa/roberta-base_topic_classification_nyt_news'
|
|
zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en'
|
|
model_params type=typing.Dict, default={}
|
|
zh_to_en_model_params type=typing.Dict, default={}
|
|
label_key type=<class 'str'>, default='query_topic_label'
|
|
score_key type=<class 'str'>, default='query_topic_label_score'
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: random_selector
|
|
【算子类】: RandomSelector
|
|
【算子类型】: Selector
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Randomly selects a subset of samples from the dataset.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
select_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None
|
|
select_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: range_specified_field_selector
|
|
【算子类】: RangeSpecifiedFieldSelector
|
|
【算子类型】: Selector
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Selects a range of samples based on the sorted values of a specified field.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
lower_percentile type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None
|
|
upper_percentile type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None
|
|
lower_rank type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
upper_rank type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: ray_bts_minhash_deduplicator
|
|
【算子类】: RayBTSMinhashDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: A MinhashLSH deduplicator that operates in Ray distributed mode.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tokenization type=<class 'str'>, default='space'
|
|
window_size type=typing.Annotated[int, Gt(gt=0)], default=5
|
|
lowercase type=<class 'bool'>, default=True
|
|
ignore_pattern type=typing.Optional[str], default=None
|
|
num_permutations type=typing.Annotated[int, Gt(gt=0)], default=256
|
|
jaccard_threshold type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.7
|
|
num_bands type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
num_rows_per_band type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
tokenizer_model type=typing.Optional[str], default=None
|
|
union_find_parallel_num type=typing.Union[int, str], default='auto'
|
|
union_threshold type=typing.Optional[int], default=256
|
|
max_pending_edge_buffer_task type=typing.Optional[int], default=20
|
|
num_edge_buffer_task_returns type=typing.Optional[int], default=10
|
|
max_pending_filter_tasks type=typing.Optional[int], default=20
|
|
num_filter_task_returns type=typing.Optional[int], default=10
|
|
merge_batch_size type=typing.Optional[int], default=1000
|
|
minhash_batch_size type=typing.Union[str, int, NoneType], default='auto'
|
|
memory_per_sample type=typing.Optional[float], default=0.1
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: ray_document_deduplicator
|
|
【算子类】: RayDocumentDeduplicator
|
|
【算子类型】: RayBasicDeduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using exact matching in Ray distributed mode.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
backend type=<class 'str'>, default='ray_actor'
|
|
redis_address type=<class 'str'>, default='redis://localhost:6379'
|
|
lowercase type=<class 'bool'>, default=False
|
|
ignore_non_character type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: ray_image_deduplicator
|
|
【算子类】: RayImageDeduplicator
|
|
【算子类型】: RayBasicDeduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using exact matching of images in Ray distributed mode.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
backend type=<class 'str'>, default='ray_actor'
|
|
redis_address type=<class 'str'>, default='redis://localhost:6379'
|
|
method type=<class 'str'>, default='phash'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: ray_video_deduplicator
|
|
【算子类】: RayVideoDeduplicator
|
|
【算子类型】: RayBasicDeduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at document-level using exact matching of videos in Ray distributed mode.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
backend type=<class 'str'>, default='ray_actor'
|
|
redis_address type=<class 'str'>, default='redis://localhost:6379'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: relation_identity_mapper
|
|
【算子类】: RelationIdentityMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Identify the relation between two entities in a given text.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_model type=<class 'str'>, default='gpt-4o'
|
|
source_entity type=<class 'str'>, default=None
|
|
target_entity type=<class 'str'>, default=None
|
|
output_key type=<class 'str'>, default='role_relation'
|
|
api_endpoint type=typing.Optional[str], default=None
|
|
response_path type=typing.Optional[str], default=None
|
|
system_prompt_template type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
output_pattern_template type=typing.Optional[str], default=None
|
|
try_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
drop_text type=<class 'bool'>, default=False
|
|
model_params type=typing.Dict, default={}
|
|
sampling_params type=typing.Dict, default={}
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_bibliography_mapper
|
|
【算子类】: RemoveBibliographyMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Removes bibliography sections at the end of LaTeX documents.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_comments_mapper
|
|
【算子类】: RemoveCommentsMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Removes comments from documents, currently supporting only 'tex' format.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
doc_type type=typing.Union[str, typing.List[str]], default='tex'
|
|
inline type=<class 'bool'>, default=True
|
|
multiline type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_header_mapper
|
|
【算子类】: RemoveHeaderMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Removes headers at the beginning of documents in LaTeX samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
drop_no_head type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_long_words_mapper
|
|
【算子类】: RemoveLongWordsMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to remove long words within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_len type=<class 'int'>, default=1
|
|
max_len type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_non_chinese_character_mapper
|
|
【算子类】: RemoveNonChineseCharacterlMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Removes non-Chinese characters from text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
keep_alphabet type=<class 'bool'>, default=True
|
|
keep_number type=<class 'bool'>, default=True
|
|
keep_punc type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_repeat_sentences_mapper
|
|
【算子类】: RemoveRepeatSentencesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to remove repeat sentences in text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lowercase type=<class 'bool'>, default=False
|
|
ignore_special_character type=<class 'bool'>, default=True
|
|
min_repeat_sentence_length type=<class 'int'>, default=2
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_specific_chars_mapper
|
|
【算子类】: RemoveSpecificCharsMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Removes specific characters from text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
chars_to_remove type=typing.Union[str, typing.List[str]], default='◆●■►▼▲▴∆▻▷❖♡□'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_table_text_mapper
|
|
【算子类】: RemoveTableTextMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to remove table texts from text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_col type=typing.Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=2), Le(le=20)])], default=2
|
|
max_col type=typing.Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=2), Le(le=20)])], default=20
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: remove_words_with_incorrect_substrings_mapper
|
|
【算子类】: RemoveWordsWithIncorrectSubstringsMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to remove words containing specified incorrect substrings.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
tokenization type=<class 'bool'>, default=False
|
|
substrings type=typing.Optional[typing.List[str]], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: replace_content_mapper
|
|
【算子类】: ReplaceContentMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Replaces content in the text that matches a specific regular expression pattern with a
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
pattern type=typing.Union[str, typing.List[str], NoneType], default=None
|
|
repl type=typing.Union[str, typing.List[str]], default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: sdxl_prompt2prompt_mapper
|
|
【算子类】: SDXLPrompt2PromptMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates pairs of similar images using the SDXL model.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_diffusion type=<class 'str'>, default='stabilityai/stable-diffusion-xl-base-1.0'
|
|
trust_remote_code type=<class 'inspect._empty'>, default=False
|
|
torch_dtype type=<class 'str'>, default='fp32'
|
|
num_inference_steps type=<class 'float'>, default=50
|
|
guidance_scale type=<class 'float'>, default=7.5
|
|
text_key type=<class 'inspect._empty'>, default=None
|
|
text_key_second type=<class 'inspect._empty'>, default=None
|
|
output_dir type=<class 'inspect._empty'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: sentence_augmentation_mapper
|
|
【算子类】: SentenceAugmentationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Augments sentences by generating enhanced versions using a Hugging Face model. This
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_model type=<class 'str'>, default='Qwen/Qwen2-7B-Instruct'
|
|
system_prompt type=<class 'str'>, default=None
|
|
task_sentence type=<class 'str'>, default=None
|
|
max_new_tokens type=<class 'inspect._empty'>, default=256
|
|
temperature type=<class 'inspect._empty'>, default=0.2
|
|
top_p type=<class 'inspect._empty'>, default=None
|
|
num_beams type=<class 'inspect._empty'>, default=1
|
|
text_key type=<class 'inspect._empty'>, default=None
|
|
text_key_second type=<class 'inspect._empty'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: sentence_split_mapper
|
|
【算子类】: SentenceSplitMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Splits text samples into individual sentences based on the specified language.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: special_characters_filter
|
|
【算子类】: SpecialCharactersFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with special-character ratio within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_ratio type=<class 'float'>, default=0.0
|
|
max_ratio type=<class 'float'>, default=0.25
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: specified_field_filter
|
|
【算子类】: SpecifiedFieldFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter samples based on the specified field information.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
target_value type=typing.List, default=[]
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: specified_numeric_field_filter
|
|
【算子类】: SpecifiedNumericFieldFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter samples based on a specified numeric field value.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
min_value type=<class 'float'>, default=-9223372036854775807
|
|
max_value type=<class 'float'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: stopwords_filter
|
|
【算子类】: StopWordsFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with stopword ratio within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
tokenization type=<class 'bool'>, default=False
|
|
min_ratio type=<class 'float'>, default=0.3
|
|
max_ratio type=<class 'float'>, default=1.0
|
|
stopwords_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
use_words_aug type=<class 'bool'>, default=False
|
|
words_aug_group_sizes type=typing.List[typing.Annotated[int, Gt(gt=0)]], default=[2]
|
|
words_aug_join_char type=<class 'str'>, default=''
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: suffix_filter
|
|
【算子类】: SuffixFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with specified suffix.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
suffixes type=typing.Union[str, typing.List[str]], default=[]
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: tags_specified_field_selector
|
|
【算子类】: TagsSpecifiedFieldSelector
|
|
【算子类型】: Selector
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Selector to filter samples based on the tags of a specified field.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
target_tags type=typing.List[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_action_filter
|
|
【算子类】: TextActionFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep texts that contain a minimum number of actions.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
min_action_num type=<class 'int'>, default=1
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_chunk_mapper
|
|
【算子类】: TextChunkMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Split input text into chunks based on specified criteria.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
max_len type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
split_pattern type=typing.Optional[str], default='\\n\\n'
|
|
overlap_len type=typing.Annotated[int, Ge(ge=0)], default=0
|
|
tokenizer type=typing.Optional[str], default=None
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_embd_similarity_filter
|
|
【算子类】: TextEmbdSimilarityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep texts whose average embedding similarity to a set of given validation
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
api_or_hf_model type=<class 'str'>, default='text-embedding-v4'
|
|
is_hf_model type=<class 'bool'>, default=False
|
|
api_endpoint type=<class 'str'>, default='embeddings'
|
|
response_path type=<class 'str'>, default='data.0.embedding'
|
|
model_params type=typing.Optional[typing.Dict], default=None
|
|
min_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=0.1
|
|
max_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=1.0
|
|
valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None
|
|
ebd_dim type=<class 'int'>, default=4096
|
|
pooling type=typing.Optional[str], default=None
|
|
input_template type=typing.Optional[str], default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_entity_dependency_filter
|
|
【算子类】: TextEntityDependencyFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Identify and filter text samples based on entity dependencies.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
min_dependency_num type=<class 'int'>, default=1
|
|
any_or_all type=<class 'str'>, default='all'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_length_filter
|
|
【算子类】: TextLengthFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with total text length within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_len type=<class 'int'>, default=10
|
|
max_len type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: text_pair_similarity_filter
|
|
【算子类】: TextPairSimilarityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep text pairs with similarities within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_clip type=<class 'inspect._empty'>, default='openai/clip-vit-base-patch32'
|
|
trust_remote_code type=<class 'inspect._empty'>, default=False
|
|
min_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=0.1
|
|
max_score type=<class 'jsonargparse.typing.ClosedUnitInterval'>, default=1.0
|
|
text_key_second type=<class 'inspect._empty'>, default=None
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: token_num_filter
|
|
【算子类】: TokenNumFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with a total token number within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_tokenizer type=<class 'str'>, default='EleutherAI/pythia-6.9b-deduped'
|
|
min_num type=<class 'int'>, default=10
|
|
max_num type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: topk_specified_field_selector
|
|
【算子类】: TopkSpecifiedFieldSelector
|
|
【算子类型】: Selector
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Selects top samples based on the sorted values of a specified field.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
field_key type=<class 'str'>, default=''
|
|
top_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None
|
|
topk type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
reverse type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: vggt_mapper
|
|
【算子类】: VggtMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Input a video of a single scene, and use VGGT to extract information including Camera
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
vggt_model_path type=<class 'str'>, default='facebook/VGGT-1B'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
duration type=<class 'float'>, default=0
|
|
tag_field_name type=<class 'str'>, default='vggt_tags'
|
|
frame_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
if_output_camera_parameters type=<class 'bool'>, default=True
|
|
if_output_depth_maps type=<class 'bool'>, default=True
|
|
if_output_point_maps_from_projection type=<class 'bool'>, default=True
|
|
if_output_point_maps_from_unprojection type=<class 'bool'>, default=True
|
|
if_output_point_tracks type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_aesthetics_filter
|
|
【算子类】: VideoAestheticsFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep data samples with aesthetics scores for specified frames in the videos
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_scorer_model type=<class 'str'>, default=''
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.4
|
|
max_score type=<class 'float'>, default=1.0
|
|
frame_sampling_method type=<class 'str'>, default='uniform'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
any_or_all type=<class 'str'>, default='any'
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_aspect_ratio_filter
|
|
【算子类】: VideoAspectRatioFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with video aspect ratio within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_ratio type=<class 'str'>, default='9/21'
|
|
max_ratio type=<class 'str'>, default='21/9'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_captioning_from_audio_mapper
|
|
【算子类】: VideoCaptioningFromAudioMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to caption a video according to its audio streams based on
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_captioning_from_frames_mapper
|
|
【算子类】: VideoCaptioningFromFramesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Generates video captions from sampled frames using an image-to-text model. Captions from
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_img2seq type=<class 'str'>, default='Salesforce/blip2-opt-2.7b'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
caption_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_candidate_mode type=<class 'str'>, default='random_any'
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
prompt type=typing.Optional[str], default=None
|
|
prompt_key type=typing.Optional[str], default=None
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_captioning_from_summarizer_mapper
|
|
【算子类】: VideoCaptioningFromSummarizerMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to generate video captions by summarizing several kinds of generated
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_summarizer type=<class 'str'>, default=None
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
consider_video_caption_from_video type=<class 'bool'>, default=True
|
|
consider_video_caption_from_audio type=<class 'bool'>, default=True
|
|
consider_video_caption_from_frames type=<class 'bool'>, default=True
|
|
consider_video_tags_from_audio type=<class 'bool'>, default=True
|
|
consider_video_tags_from_frames type=<class 'bool'>, default=True
|
|
vid_cap_from_vid_args type=typing.Optional[typing.Dict], default=None
|
|
vid_cap_from_frm_args type=typing.Optional[typing.Dict], default=None
|
|
vid_tag_from_aud_args type=typing.Optional[typing.Dict], default=None
|
|
vid_tag_from_frm_args type=typing.Optional[typing.Dict], default=None
|
|
keep_tag_num type=typing.Annotated[int, Gt(gt=0)], default=5
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_captioning_from_video_mapper
|
|
【算子类】: VideoCaptioningFromVideoMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: True
|
|
【文档说明】: Generates video captions using a Hugging Face video-to-text model and sampled video
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_video_blip type=<class 'str'>, default='kpyu/video-blip-opt-2.7b-ego4d'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
caption_num type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
keep_candidate_mode type=<class 'str'>, default='random_any'
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
prompt type=typing.Optional[str], default=None
|
|
prompt_key type=typing.Optional[str], default=None
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_deduplicator
|
|
【算子类】: VideoDeduplicator
|
|
【算子类型】: Deduplicator
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Deduplicates samples at the document level using exact matching of videos.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
consider_text type=<class 'bool'>, default=False
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_duration_filter
|
|
【算子类】: VideoDurationFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples whose videos' durations are within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_duration type=<class 'float'>, default=0
|
|
max_duration type=<class 'float'>, default=9223372036854775807
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_extract_frames_mapper
|
|
【算子类】: VideoExtractFramesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Mapper to extract frames from video files according to specified methods.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
output_format type=<class 'str'>, default='path'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
duration type=<class 'float'>, default=0
|
|
frame_dir type=<class 'str'>, default=None
|
|
frame_key type=<class 'str'>, default='video_frames'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_face_blur_mapper
|
|
【算子类】: VideoFaceBlurMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Mapper to blur faces detected in videos.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
cv_classifier type=<class 'str'>, default=''
|
|
blur_type type=<class 'str'>, default='gaussian'
|
|
radius type=<class 'float'>, default=2
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_ffmpeg_wrapped_mapper
|
|
【算子类】: VideoFFmpegWrappedMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Wraps FFmpeg video filters for processing video files in a dataset.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
filter_name type=typing.Optional[str], default=None
|
|
filter_kwargs type=typing.Optional[typing.Dict], default=None
|
|
global_args type=typing.Optional[typing.List[str]], default=None
|
|
capture_stderr type=<class 'bool'>, default=True
|
|
overwrite_output type=<class 'bool'>, default=True
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_frames_text_similarity_filter
|
|
【算子类】: VideoFramesTextSimilarityFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples based on the similarity between video frame images and text
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_clip type=<class 'inspect._empty'>, default='openai/clip-vit-base-patch32'
|
|
trust_remote_code type=<class 'inspect._empty'>, default=False
|
|
min_score type=<class 'float'>, default=0.1
|
|
max_score type=<class 'float'>, default=1.0
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
horizontal_flip type=<class 'bool'>, default=False
|
|
vertical_flip type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_hand_reconstruction_mapper
|
|
【算子类】: VideoHandReconstructionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Use the WiLoR model for hand localization and
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
wilor_model_path type=<class 'str'>, default='wilor_final.ckpt'
|
|
wilor_model_config type=<class 'str'>, default='model_config.yaml'
|
|
detector_model_path type=<class 'str'>, default='detector.pt'
|
|
mano_right_path type=<class 'str'>, default='path_to_mano_right_pkl'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
duration type=<class 'float'>, default=0
|
|
batch_size type=<class 'int'>, default=16
|
|
tag_field_name type=<class 'str'>, default='hand_reconstruction_tags'
|
|
frame_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
if_save_visualization type=<class 'bool'>, default=True
|
|
save_visualization_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
if_save_mesh type=<class 'bool'>, default=True
|
|
save_mesh_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_motion_score_filter
|
|
【算子类】: VideoMotionScoreFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with video motion scores within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_score type=<class 'float'>, default=0.25
|
|
max_score type=<class 'float'>, default=1.7976931348623157e+308
|
|
sampling_fps type=typing.Annotated[float, Gt(gt=0)], default=2
|
|
size type=typing.Union[typing.Annotated[int, Gt(gt=0)], typing.Tuple[typing.Annotated[int, Gt(gt=0)]], typing.Tuple[typing.Annotated[int, Gt(gt=0)], typing.Annotated[int, Gt(gt=0)]], NoneType], default=None
|
|
max_size type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
divisible type=typing.Annotated[int, Gt(gt=0)], default=1
|
|
relative type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_motion_score_raft_filter
|
|
【算子类】: VideoMotionScoreRaftFilter
|
|
【算子类型】: VideoMotionScoreFilter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples with video motion scores within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_score type=<class 'float'>, default=1.0
|
|
max_score type=<class 'float'>, default=1.7976931348623157e+308
|
|
sampling_fps type=typing.Annotated[float, Gt(gt=0)], default=2
|
|
size type=typing.Union[typing.Annotated[int, Gt(gt=0)], typing.Tuple[typing.Annotated[int, Gt(gt=0)]], typing.Tuple[typing.Annotated[int, Gt(gt=0)], typing.Annotated[int, Gt(gt=0)]], NoneType], default=None
|
|
max_size type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None
|
|
divisible type=typing.Annotated[int, Gt(gt=0)], default=8
|
|
relative type=<class 'bool'>, default=False
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_nsfw_filter
|
|
【算子类】: VideoNSFWFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples whose videos have nsfw scores in a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_nsfw_model type=<class 'str'>, default='Falconsai/nsfw_image_detection'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
min_score type=<class 'float'>, default=0.0
|
|
max_score type=<class 'float'>, default=0.5
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_ocr_area_ratio_filter
|
|
【算子类】: VideoOcrAreaRatioFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples whose detected text area ratios for specified frames in the video are
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_area_ratio type=<class 'float'>, default=0
|
|
max_area_ratio type=<class 'float'>, default=1.0
|
|
frame_sample_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
languages_to_detect type=typing.Union[str, typing.List[str]], default=['ch_sim', 'en']
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_remove_watermark_mapper
|
|
【算子类】: VideoRemoveWatermarkMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Remove watermarks from videos based on specified regions.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
roi_strings type=typing.List[str], default=['0,0,0.1,0.1']
|
|
roi_type type=<class 'str'>, default='ratio'
|
|
roi_key type=typing.Optional[str], default=None
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=10
|
|
min_frame_threshold type=typing.Annotated[int, Gt(gt=0)], default=7
|
|
detection_method type=<class 'str'>, default='pixel_value'
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_resize_aspect_ratio_mapper
|
|
【算子类】: VideoResizeAspectRatioMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Resizes videos to fit within a specified aspect ratio range. This operator adjusts the
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_ratio type=<class 'str'>, default='9/21'
|
|
max_ratio type=<class 'str'>, default='21/9'
|
|
strategy type=<class 'str'>, default='increase'
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_resize_resolution_mapper
|
|
【算子类】: VideoResizeResolutionMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Resizes video resolution based on specified width and height constraints.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_width type=<class 'int'>, default=1
|
|
max_width type=<class 'int'>, default=9223372036854775807
|
|
min_height type=<class 'int'>, default=1
|
|
max_height type=<class 'int'>, default=9223372036854775807
|
|
force_original_aspect_ratio type=<class 'str'>, default='disable'
|
|
force_divisible_by type=typing.Annotated[int, Gt(gt=0)], default=2
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_resolution_filter
|
|
【算子类】: VideoResolutionFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Keep data samples whose videos' resolutions are within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
min_width type=<class 'int'>, default=1
|
|
max_width type=<class 'int'>, default=9223372036854775807
|
|
min_height type=<class 'int'>, default=1
|
|
max_height type=<class 'int'>, default=9223372036854775807
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_split_by_duration_mapper
|
|
【算子类】: VideoSplitByDurationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Splits videos into segments based on a specified duration.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
split_duration type=<class 'float'>, default=10
|
|
min_last_split_duration type=<class 'float'>, default=0
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
save_dir type=<class 'str'>, default=None
|
|
video_backend type=<class 'str'>, default='ffmpeg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_split_by_key_frame_mapper
|
|
【算子类】: VideoSplitByKeyFrameMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Splits a video into segments based on key frames.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
keep_original_sample type=<class 'bool'>, default=True
|
|
save_dir type=<class 'str'>, default=None
|
|
video_backend type=<class 'str'>, default='ffmpeg'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_split_by_scene_mapper
|
|
【算子类】: VideoSplitBySceneMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: False
|
|
【文档说明】: Splits videos into scene clips based on detected scene changes.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
detector type=<class 'str'>, default='ContentDetector'
|
|
threshold type=typing.Annotated[float, Ge(ge=0)], default=27.0
|
|
min_scene_len type=typing.Annotated[int, Ge(ge=0)], default=15
|
|
show_progress type=<class 'bool'>, default=False
|
|
save_dir type=<class 'str'>, default=None
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_tagging_from_audio_mapper
|
|
【算子类】: VideoTaggingFromAudioMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates video tags from audio streams using the Audio Spectrogram Transformer.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_ast type=<class 'str'>, default='MIT/ast-finetuned-audioset-10-10-0.4593'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
tag_field_name type=<class 'str'>, default='video_audio_tags'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_tagging_from_frames_filter
|
|
【算子类】: VideoTaggingFromFramesFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples whose videos contain specified tags.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
tags type=typing.List[str], default=['people']
|
|
contain type=<class 'str'>, default='any'
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
tag_field_name type=<class 'str'>, default='video_frame_tags'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_tagging_from_frames_mapper
|
|
【算子类】: VideoTaggingFromFramesMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Generates video tags from frames extracted from videos.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
tag_field_name type=<class 'str'>, default='video_frame_tags'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_watermark_filter
|
|
【算子类】: VideoWatermarkFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Filter to keep samples whose videos have no watermark with high probability.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
hf_watermark_model type=<class 'str'>, default='amrul-hzz/watermark_detector'
|
|
trust_remote_code type=<class 'bool'>, default=False
|
|
prob_threshold type=<class 'float'>, default=0.8
|
|
frame_sampling_method type=<class 'str'>, default='all_keyframes'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
reduce_mode type=<class 'str'>, default='avg'
|
|
any_or_all type=<class 'str'>, default='any'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: video_whole_body_pose_estimation_mapper
|
|
【算子类】: VideoWholeBodyPoseEstimationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cuda
|
|
【批处理算子】: False
|
|
【文档说明】: Input a video containing people, and use the DWPose
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
onnx_det_model type=<class 'str'>, default='yolox_l.onnx'
|
|
onnx_pose_model type=<class 'str'>, default='dw-ll_ucoco_384.onnx'
|
|
frame_num type=typing.Annotated[int, Gt(gt=0)], default=3
|
|
duration type=<class 'float'>, default=0
|
|
tag_field_name type=<class 'str'>, default='pose_estimation_tags'
|
|
frame_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
if_save_visualization type=<class 'bool'>, default=False
|
|
save_visualization_dir type=<class 'str'>, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets'
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: whitespace_normalization_mapper
|
|
【算子类】: WhitespaceNormalizationMapper
|
|
【算子类型】: Mapper
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Normalizes various types of whitespace characters to standard spaces in text samples.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: word_repetition_filter
|
|
【算子类】: WordRepetitionFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with word-level n-gram repetition ratio within a specific range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
tokenization type=<class 'bool'>, default=False
|
|
rep_len type=typing.Annotated[int, Gt(gt=0)], default=10
|
|
min_ratio type=<class 'float'>, default=0.0
|
|
max_ratio type=<class 'float'>, default=0.5
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
【算子名称】: words_num_filter
|
|
【算子类】: WordsNumFilter
|
|
【算子类型】: Filter
|
|
【加速设备】: cpu
|
|
【批处理算子】: True
|
|
【文档说明】: Filter to keep samples with a total word count within a specified range.
|
|
|
|
【支持参数列表（YAML可直接使用）】
|
|
lang type=<class 'str'>, default='en'
|
|
tokenization type=<class 'bool'>, default=False
|
|
min_num type=<class 'int'>, default=10
|
|
max_num type=<class 'int'>, default=9223372036854775807
|
|
args type=<class 'inspect._empty'>, default=required (无默认值)
|
|
kwargs type=<class 'inspect._empty'>, default=required (无默认值)
|
|
================================================================================
|
|
|
|
(python3.11) D:\xu_projects\for_mage\llm\repos\DeepData\test\ray>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|