Total registered operators: 166 ================================================================================ 【算子名称】: alphanumeric_filter 【算子类】: AlphanumericFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with an alphabet/numeric ratio within a specific range. 【支持参数列表（YAML可直接使用）】 tokenization type=, default=False min_ratio type=, default=0.25 max_ratio type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: audio_add_gaussian_noise_mapper 【算子类】: AudioAddGaussianNoiseMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to add Gaussian noise to audio samples. 【支持参数列表（YAML可直接使用）】 min_amplitude type=, default=0.001 max_amplitude type=, default=0.015 p type=, default=0.5 save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: audio_duration_filter 【算子类】: AudioDurationFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Keep data samples whose audio durations are within a specified range. 【支持参数列表（YAML可直接使用）】 min_duration type=, default=0 max_duration type=, default=9223372036854775807 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: audio_ffmpeg_wrapped_mapper 【算子类】: AudioFFmpegWrappedMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Wraps FFmpeg audio filters for processing audio files in a dataset. 【支持参数列表（YAML可直接使用）】 filter_name type=typing.Optional[str], default=None filter_kwargs type=typing.Optional[typing.Dict], default=None global_args type=typing.Optional[typing.List[str]], default=None capture_stderr type=, default=True overwrite_output type=, default=True save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: audio_nmf_snr_filter 【算子类】: AudioNMFSNRFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Keep data samples whose audio Signal-to-Noise Ratios (SNRs) are within a specified 【支持参数列表（YAML可直接使用）】 min_snr type=, default=0 max_snr type=, default=9223372036854775807 nmf_iter_num type=typing.Annotated[int, Gt(gt=0)], default=500 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: audio_size_filter 【算子类】: AudioSizeFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Keep data samples based on the size of their audio files. 【支持参数列表（YAML可直接使用）】 min_size type=, default='0' max_size type=, default='1TB' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: average_line_length_filter 【算子类】: AverageLineLengthFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with average line length within a specific range. 【支持参数列表（YAML可直接使用）】 min_len type=, default=10 max_len type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: calibrate_qa_mapper 【算子类】: CalibrateQAMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Calibrates question-answer pairs based on reference text using an API model. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None reference_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: calibrate_query_mapper 【算子类】: CalibrateQueryMapper 【算子类型】: CalibrateQAMapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Calibrate query in question-answer pairs based on reference text. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None reference_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: calibrate_response_mapper 【算子类】: CalibrateResponseMapper 【算子类型】: CalibrateQAMapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Calibrate response in question-answer pairs based on reference text. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None reference_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: character_repetition_filter 【算子类】: CharacterRepetitionFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with character-level n-gram repetition ratio within a specific 【支持参数列表（YAML可直接使用）】 rep_len type=typing.Annotated[int, Gt(gt=0)], default=10 min_ratio type=, default=0.0 max_ratio type=, default=0.5 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: chinese_convert_mapper 【算子类】: ChineseConvertMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to convert Chinese text between Traditional, Simplified, and Japanese Kanji. 【支持参数列表（YAML可直接使用）】 mode type=, default='s2t' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: clean_copyright_mapper 【算子类】: CleanCopyrightMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Cleans copyright comments at the beginning of text samples. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: clean_email_mapper 【算子类】: CleanEmailMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Cleans email addresses from text samples using a regular expression. 【支持参数列表（YAML可直接使用）】 pattern type=typing.Optional[str], default=None repl type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: clean_html_mapper 【算子类】: CleanHtmlMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Cleans HTML code from text samples, converting HTML to plain text. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: clean_ip_mapper 【算子类】: CleanIpMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Cleans IPv4 and IPv6 addresses from text samples. 【支持参数列表（YAML可直接使用）】 pattern type=typing.Optional[str], default=None repl type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: clean_links_mapper 【算子类】: CleanLinksMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to clean links like http/https/ftp in text samples. 【支持参数列表（YAML可直接使用）】 pattern type=typing.Optional[str], default=None repl type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: detect_character_attributes_mapper 【算子类】: DetectCharacterAttributesMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Takes an image, a caption, and main character names as input to extract the characters' attributes. 【支持参数列表（YAML可直接使用）】 detect_character_locations_mapper_args type=typing.Optional[typing.Dict], default={} args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: detect_character_locations_mapper 【算子类】: DetectCharacterLocationsMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Given an image and a list of main character names, extract the bounding boxes for each present character. 【支持参数列表（YAML可直接使用）】 mllm_mapper_args type=typing.Optional[typing.Dict], default={} image_text_matching_filter_args type=typing.Optional[typing.Dict], default={} yoloe_path type=, default='yoloe-11l-seg.pt' iou_threshold type=, default=0.7 matching_score_threshold type=, default=0.4 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: detect_main_character_mapper 【算子类】: DetectMainCharacterMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Extract all main character names based on the given image and its caption. 【支持参数列表（YAML可直接使用）】 mllm_mapper_args type=typing.Optional[typing.Dict], default={} filter_min_character_num type=, default=0 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: dialog_intent_detection_mapper 【算子类】: DialogIntentDetectionMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Generates user's intent labels in a dialog by analyzing the history, query, and 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' intent_candidates type=typing.Optional[typing.List[str]], default=None max_round type=typing.Annotated[int, Ge(ge=0)], default=10 labels_key type=, default='dialog_intent_labels' analysis_key type=, default='dialog_intent_labels_analysis' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None candidate_template type=typing.Optional[str], default=None analysis_template type=typing.Optional[str], default=None labels_template type=typing.Optional[str], default=None analysis_pattern type=typing.Optional[str], default=None labels_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: dialog_sentiment_detection_mapper 【算子类】: DialogSentimentDetectionMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Generates sentiment labels and analysis for user queries in a dialog. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' sentiment_candidates type=typing.Optional[typing.List[str]], default=None max_round type=typing.Annotated[int, Ge(ge=0)], default=10 labels_key type=, default='dialog_sentiment_labels' analysis_key type=, default='dialog_sentiment_labels_analysis' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None candidate_template type=typing.Optional[str], default=None analysis_template type=typing.Optional[str], default=None labels_template type=typing.Optional[str], default=None analysis_pattern type=typing.Optional[str], default=None labels_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: dialog_sentiment_intensity_mapper 【算子类】: DialogSentimentIntensityMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to predict user's sentiment intensity in a dialog, ranging from -5 to 5. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' max_round type=typing.Annotated[int, Ge(ge=0)], default=10 intensities_key type=, default='dialog_sentiment_intensity' analysis_key type=, default='dialog_sentiment_intensity_analysis' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None analysis_template type=typing.Optional[str], default=None intensity_template type=typing.Optional[str], default=None analysis_pattern type=typing.Optional[str], default=None intensity_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: dialog_topic_detection_mapper 【算子类】: DialogTopicDetectionMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Generates user's topic labels and analysis in a dialog. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' topic_candidates type=typing.Optional[typing.List[str]], default=None max_round type=typing.Annotated[int, Ge(ge=0)], default=10 labels_key type=, default='dialog_topic_labels' analysis_key type=, default='dialog_topic_labels_analysis' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None candidate_template type=typing.Optional[str], default=None analysis_template type=typing.Optional[str], default=None labels_template type=typing.Optional[str], default=None analysis_pattern type=typing.Optional[str], default=None labels_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: document_deduplicator 【算子类】: DocumentDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using exact matching. 【支持参数列表（YAML可直接使用）】 lowercase type=, default=False ignore_non_character type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: document_minhash_deduplicator 【算子类】: DocumentMinhashDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using MinHash LSH. 【支持参数列表（YAML可直接使用）】 tokenization type=, default='space' window_size type=typing.Annotated[int, Gt(gt=0)], default=5 lowercase type=, default=True ignore_pattern type=typing.Optional[str], default=None num_permutations type=typing.Annotated[int, Gt(gt=0)], default=256 jaccard_threshold type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.7 num_bands type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None num_rows_per_band type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None tokenizer_model type=typing.Optional[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: document_simhash_deduplicator 【算子类】: DocumentSimhashDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using SimHash. 【支持参数列表（YAML可直接使用）】 tokenization type=, default='space' window_size type=typing.Annotated[int, Gt(gt=0)], default=6 lowercase type=, default=True ignore_pattern type=typing.Optional[str], default=None num_blocks type=typing.Annotated[int, Gt(gt=0)], default=6 hamming_distance type=typing.Annotated[int, Gt(gt=0)], default=4 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: download_file_mapper 【算子类】: DownloadFileMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to download URL files to local files or load them into memory. 【支持参数列表（YAML可直接使用）】 download_field type=, default=None save_dir type=, default=None save_field type=, default=None resume_download type=, default=False timeout type=, default=30 max_concurrent type=, default=10 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: entity_attribute_aggregator 【算子类】: EntityAttributeAggregator 【算子类型】: Aggregator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Summarizes a given attribute of an entity from a set of documents. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' entity type=, default=None attribute type=, default=None input_key type=, default='event_description' output_key type=, default='entity_attribute' word_limit type=typing.Annotated[int, Gt(gt=0)], default=100 max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt_template type=typing.Optional[str], default=None example_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: expand_macro_mapper 【算子类】: ExpandMacroMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Expands macro definitions in the document body of LaTeX samples. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_entity_attribute_mapper 【算子类】: ExtractEntityAttributeMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts attributes for given entities from the text and stores them in the sample's 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' query_entities type=typing.List[str], default=[] query_attributes type=typing.List[str], default=[] entity_key type=, default='main_entities' attribute_key type=, default='attributes' attribute_desc_key type=, default='attribute_descriptions' support_text_key type=, default='attribute_support_texts' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt_template type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None attr_pattern_template type=typing.Optional[str], default=None demo_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_entity_relation_mapper 【算子类】: ExtractEntityRelationMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts entities and relations from text to build a knowledge graph. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' entity_types type=typing.List[str], default=None entity_key type=, default='entity' relation_key type=, default='relation' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None prompt_template type=typing.Optional[str], default=None tuple_delimiter type=typing.Optional[str], default=None record_delimiter type=typing.Optional[str], default=None completion_delimiter type=typing.Optional[str], default=None max_gleaning type=typing.Annotated[int, Ge(ge=0)], default=1 continue_prompt type=typing.Optional[str], default=None if_loop_prompt type=typing.Optional[str], default=None entity_pattern type=typing.Optional[str], default=None relation_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_event_mapper 【算子类】: ExtractEventMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Extracts events and relevant characters from the text. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' event_desc_key type=, default='event_description' relevant_char_key type=, default='relevant_characters' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_keyword_mapper 【算子类】: ExtractKeywordMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Generate keywords for the text. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' keyword_key type=, default='keyword' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None prompt_template type=typing.Optional[str], default=None completion_delimiter type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_nickname_mapper 【算子类】: ExtractNicknameMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts nickname relationships in the text using a language model. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' nickname_key type=, default='nickname' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_support_text_mapper 【算子类】: ExtractSupportTextMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts a supporting sub-text from the original text based on a given summary. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' summary_key type=, default='event_description' support_text_key type=, default='support_text' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: extract_tables_from_html_mapper 【算子类】: ExtractTablesFromHtmlMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts tables from HTML content and stores them in a specified field. 【支持参数列表（YAML可直接使用）】 tables_field_name type=, default='html_tables' retain_html_tags type=, default=False include_header type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: fix_unicode_mapper 【算子类】: FixUnicodeMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Fixes unicode errors in text samples. 【支持参数列表（YAML可直接使用）】 normalization type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: flagged_words_filter 【算子类】: FlaggedWordFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with flagged-word ratio in a specified range. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' tokenization type=, default=False min_ratio type=, default=0.0 max_ratio type=, default=0.045 flagged_words_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' use_words_aug type=, default=False words_aug_group_sizes type=typing.List[typing.Annotated[int, Gt(gt=0)]], default=[2] words_aug_join_char type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: frequency_specified_field_selector 【算子类】: FrequencySpecifiedFieldSelector 【算子类型】: Selector 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Selector to filter samples based on the frequency of a specified field. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' top_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None topk type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None reverse type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: general_field_filter 【算子类】: GeneralFieldFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples based on a general field filter condition. 【支持参数列表（YAML可直接使用）】 filter_condition type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: general_fused_op 【算子类】: GeneralFusedOP 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: An explicitly fused operator designed to execute multiple sequential operations (OPs) on 【支持参数列表（YAML可直接使用）】 batch_size type=, default=1 fused_op_list type=typing.Optional[typing.List], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: generate_qa_from_examples_mapper 【算子类】: GenerateQAFromExamplesMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates question and answer pairs from examples using a Hugging Face model. 【支持参数列表（YAML可直接使用）】 hf_model type=, default='Qwen/Qwen2.5-7B-Instruct' seed_file type=, default='' example_num type=typing.Annotated[int, Gt(gt=0)], default=3 similarity_threshold type=, default=0.7 system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None example_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None enable_vllm type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: generate_qa_from_text_mapper 【算子类】: GenerateQAFromTextMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Generates question and answer pairs from text using a specified model. 【支持参数列表（YAML可直接使用）】 hf_model type=, default='alibaba-pai/pai-qwen1_5-7b-doc2qa' max_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None output_pattern type=typing.Optional[str], default=None enable_vllm type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: human_preference_annotation_mapper 【算子类】: HumanPreferenceAnnotationMapper 【算子类型】: LabelStudioAnnotationMapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Operator for human preference annotation using Label Studio. 【支持参数列表（YAML可直接使用）】 label_config_file type=, default=None answer1_key type=, default='answer1' answer2_key type=, default='answer2' prompt_key type=, default='prompt' chosen_key type=, default='chosen' rejected_key type=, default='rejected' kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_aesthetics_filter 【算子类】: ImageAestheticsFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with aesthetics scores within a specific range. 【支持参数列表（YAML可直接使用）】 hf_scorer_model type=, default='' trust_remote_code type=, default=False min_score type=, default=0.5 max_score type=, default=1.0 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_aspect_ratio_filter 【算子类】: ImageAspectRatioFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with image aspect ratio within a specific range. 【支持参数列表（YAML可直接使用）】 min_ratio type=, default=0.333 max_ratio type=, default=3.0 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_blur_mapper 【算子类】: ImageBlurMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Blurs images in the dataset with a specified probability and blur type. 【支持参数列表（YAML可直接使用）】 p type=, default=0.2 blur_type type=, default='gaussian' radius type=, default=2 save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_captioning_from_gpt4v_mapper 【算子类】: ImageCaptioningFromGPT4VMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Generates text captions for images using the GPT-4 Vision model. 【支持参数列表（YAML可直接使用）】 mode type=, default='description' api_key type=, default='' max_token type=, default=500 temperature type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=1.0 system_prompt type=, default='' user_prompt type=, default='' user_prompt_key type=typing.Optional[str], default=None keep_original_sample type=, default=True any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_captioning_mapper 【算子类】: ImageCaptioningMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Generates image captions using a Hugging Face model and appends them to samples. 【支持参数列表（YAML可直接使用）】 hf_img2seq type=, default='Salesforce/blip2-opt-2.7b' trust_remote_code type=, default=False caption_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_candidate_mode type=, default='random_any' keep_original_sample type=, default=True prompt type=typing.Optional[str], default=None prompt_key type=typing.Optional[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_deduplicator 【算子类】: ImageDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level by exact matching of images. 【支持参数列表（YAML可直接使用）】 method type=, default='phash' consider_text type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_detection_yolo_mapper 【算子类】: ImageDetectionYoloMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Perform object detection using YOLO on images and return bounding boxes and class 【支持参数列表（YAML可直接使用）】 imgsz type=, default=640 conf type=, default=0.05 iou type=, default=0.5 model_path type=, default='yolo11n.pt' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_diffusion_mapper 【算子类】: ImageDiffusionMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Generate images using a diffusion model based on provided captions. 【支持参数列表（YAML可直接使用）】 hf_diffusion type=, default='CompVis/stable-diffusion-v1-4' trust_remote_code type=, default=False torch_dtype type=, default='fp32' revision type=, default='main' strength type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.8 guidance_scale type=, default=7.5 aug_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_original_sample type=, default=True caption_key type=typing.Optional[str], default=None hf_img2seq type=, default='Salesforce/blip2-opt-2.7b' save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_face_blur_mapper 【算子类】: ImageFaceBlurMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to blur faces detected in images. 【支持参数列表（YAML可直接使用）】 cv_classifier type=, default='' blur_type type=, default='gaussian' radius type=typing.Annotated[float, Ge(ge=0)], default=2 save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_face_count_filter 【算子类】: ImageFaceCountFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with the number of faces within a specific range. 【支持参数列表（YAML可直接使用）】 cv_classifier type=, default='' min_face_count type=, default=1 max_face_count type=, default=1 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_face_ratio_filter 【算子类】: ImageFaceRatioFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with face area ratios within a specific range. 【支持参数列表（YAML可直接使用）】 cv_classifier type=, default='' min_ratio type=, default=0.0 max_ratio type=, default=0.4 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_nsfw_filter 【算子类】: ImageNSFWFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples whose images have nsfw scores in a specified range. 【支持参数列表（YAML可直接使用）】 hf_nsfw_model type=, default='Falconsai/nsfw_image_detection' trust_remote_code type=, default=False min_score type=, default=0.0 max_score type=, default=0.5 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_pair_similarity_filter 【算子类】: ImagePairSimilarityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep image pairs with similarities between images within a specific range. 【支持参数列表（YAML可直接使用）】 hf_clip type=, default='openai/clip-vit-base-patch32' trust_remote_code type=, default=False min_score type=, default=0.1 max_score type=, default=1.0 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_remove_background_mapper 【算子类】: ImageRemoveBackgroundMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to remove the background of images. 【支持参数列表（YAML可直接使用）】 alpha_matting type=, default=False alpha_matting_foreground_threshold type=, default=240 alpha_matting_background_threshold type=, default=10 alpha_matting_erode_size type=, default=10 bgcolor type=typing.Optional[typing.Tuple[int, int, int, int]], default=None save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_segment_mapper 【算子类】: ImageSegmentMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Perform segment-anything on images and return the bounding boxes. 【支持参数列表（YAML可直接使用）】 imgsz type=, default=1024 conf type=, default=0.05 iou type=, default=0.5 model_path type=, default='FastSAM-x.pt' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_shape_filter 【算子类】: ImageShapeFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with image shape (width, height) within specific ranges. 【支持参数列表（YAML可直接使用）】 min_width type=, default=1 max_width type=, default=9223372036854775807 min_height type=, default=1 max_height type=, default=9223372036854775807 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_size_filter 【算子类】: ImageSizeFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Keep data samples whose image size (in Bytes/KB/MB/...) is within a specific range. 【支持参数列表（YAML可直接使用）】 min_size type=, default='0' max_size type=, default='1TB' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_tagging_mapper 【算子类】: ImageTaggingMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates image tags for each image in the sample. 【支持参数列表（YAML可直接使用）】 tag_field_name type=, default='image_tags' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_text_matching_filter 【算子类】: ImageTextMatchingFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with image-text matching scores within a specific range. 【支持参数列表（YAML可直接使用）】 hf_blip type=, default='Salesforce/blip-itm-base-coco' trust_remote_code type=, default=False min_score type=, default=0.003 max_score type=, default=1.0 horizontal_flip type=, default=False vertical_flip type=, default=False any_or_all type=, default='any' reduce_mode type=, default='avg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_text_similarity_filter 【算子类】: ImageTextSimilarityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Filter to keep samples with image-text similarity within a specified range. 【支持参数列表（YAML可直接使用）】 hf_clip type=, default='openai/clip-vit-base-patch32' trust_remote_code type=, default=False min_score type=, default=0.1 max_score type=, default=1.0 horizontal_flip type=, default=False vertical_flip type=, default=False any_or_all type=, default='any' reduce_mode type=, default='avg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: image_watermark_filter 【算子类】: ImageWatermarkFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples whose images have no watermark with high probability. 【支持参数列表（YAML可直接使用）】 hf_watermark_model type=, default='amrul-hzz/watermark_detector' trust_remote_code type=, default=False prob_threshold type=, default=0.8 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: imgdiff_difference_area_generator_mapper 【算子类】: Difference_Area_Generator_Mapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates and filters bounding boxes for image pairs based on similarity, segmentation, 【支持参数列表（YAML可直接使用）】 image_pair_similarity_filter_args type=typing.Optional[typing.Dict], default={} image_segment_mapper_args type=typing.Optional[typing.Dict], default={} image_text_matching_filter_args type=typing.Optional[typing.Dict], default={} args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: imgdiff_difference_caption_generator_mapper 【算子类】: Difference_Caption_Generator_Mapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates difference captions for bounding box regions in two images. 【支持参数列表（YAML可直接使用）】 mllm_mapper_args type=typing.Optional[typing.Dict], default={} image_text_matching_filter_args type=typing.Optional[typing.Dict], default={} text_pair_similarity_filter_args type=typing.Optional[typing.Dict], default={} args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: in_context_influence_filter 【算子类】: InContextInfluenceFilter 【算子类型】: LLMPerplexityFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep texts based on their in-context influence on a validation set. 【支持参数列表（YAML可直接使用）】 valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None task_desc type=, default=None valid_as_demo type=, default=False n_shot type=typing.Optional[int], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: instruction_following_difficulty_filter 【算子类】: InstructionFollowingDifficultyFilter 【算子类型】: LLMPerplexityFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep texts based on their instruction following difficulty (IFD, 【支持参数列表（YAML可直接使用）】 hf_model type=, default='Qwen/Qwen2.5-0.5B' model_params type=typing.Optional[typing.Dict], default=None min_score type=, default=1.0 max_score type=, default=100.0 query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: key_value_grouper 【算子类】: KeyValueGrouper 【算子类型】: Grouper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Groups samples into batches based on values in specified keys. 【支持参数列表（YAML可直接使用）】 group_by_keys type=typing.Optional[typing.List[str]], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: language_id_score_filter 【算子类】: LanguageIDScoreFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples in a specific language with a confidence score above a threshold. 【支持参数列表（YAML可直接使用）】 lang type=typing.Union[str, typing.List[str]], default='' min_score type=, default=0.8 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: llm_analysis_filter 【算子类】: LLMAnalysisFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Base filter class for leveraging LLMs to analyze and filter data samples. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='gpt-4o' min_score type=, default=0.5 max_score type=, default=1.0 is_hf_model type=, default=False api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None input_keys type=typing.List[str], default=['text'] field_names type=typing.List[str], default=['Text'] system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None field_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} dim_required_keys type=typing.Optional[typing.List[str]], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: llm_difficulty_score_filter 【算子类】: LLMDifficultyScoreFilter 【算子类型】: LLMAnalysisFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with high difficulty scores estimated by an LLM. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='gpt-4o' min_score type=, default=0.5 max_score type=, default=1.0 is_hf_model type=, default=False api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None input_keys type=typing.List[str], default=['text'] field_names type=typing.List[str], default=['Text'] system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None field_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} dim_required_keys type=typing.Optional[typing.List[str]], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: llm_perplexity_filter 【算子类】: LLMPerplexityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with perplexity scores within a specified range, computed using a 【支持参数列表（YAML可直接使用）】 hf_model type=, default='Qwen/Qwen2.5-0.5B' model_params type=typing.Optional[typing.Dict], default=None min_score type=, default=1.0 max_score type=, default=100.0 query_template type=typing.Optional[str], default=None response_template type=typing.Optional[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: llm_quality_score_filter 【算子类】: LLMQualityScoreFilter 【算子类型】: LLMAnalysisFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with a high quality score estimated by a language model. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='gpt-4o' min_score type=, default=0.5 max_score type=, default=1.0 is_hf_model type=, default=False api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None input_keys type=typing.List[str], default=['text'] field_names type=typing.List[str], default=['Text'] system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None field_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} dim_required_keys type=typing.Optional[typing.List[str]], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: llm_task_relevance_filter 【算子类】: LLMTaskRelevanceFilter 【算子类型】: LLMAnalysisFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with high relevance scores to validation tasks estimated by an 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='gpt-4o' min_score type=, default=0.5 is_hf_model type=, default=False valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None task_desc type=typing.Optional[str], default=None n_shot type=typing.Optional[int], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: maximum_line_length_filter 【算子类】: MaximumLineLengthFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with a maximum line length within a specified range. 【支持参数列表（YAML可直接使用）】 min_len type=, default=10 max_len type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: meta_tags_aggregator 【算子类】: MetaTagsAggregator 【算子类型】: Aggregator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Merge similar meta tags into a single, unified tag. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' meta_tag_key type=, default='dialog_sentiment_labels' target_tags type=typing.Optional[typing.List[str]], default=None api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None target_tag_template type=typing.Optional[str], default=None tag_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: mllm_mapper 【算子类】: MllmMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Mapper to use MLLMs for visual question answering tasks. This operator uses a Hugging 【支持参数列表（YAML可直接使用）】 hf_model type=, default='llava-hf/llava-v1.6-vicuna-7b-hf' max_new_tokens type=, default=256 temperature type=, default=0.2 top_p type=, default=None num_beams type=, default=1 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: most_relevant_entities_aggregator 【算子类】: MostRelevantEntitiesAggregator 【算子类型】: Aggregator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Extracts and ranks entities closely related to a given entity from provided texts. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' entity type=, default=None query_entity_type type=, default=None input_key type=, default='event_description' output_key type=, default='most_relevant_entities' max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt_template type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: naive_grouper 【算子类】: NaiveGrouper 【算子类型】: Grouper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Group all samples in a dataset into a single batched sample. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: naive_reverse_grouper 【算子类】: NaiveReverseGrouper 【算子类型】: Grouper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Split batched samples into individual samples. 【支持参数列表（YAML可直接使用）】 batch_meta_export_path type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: nested_aggregator 【算子类】: NestedAggregator 【算子类型】: Aggregator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Aggregates nested content from multiple samples into a single summary. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' input_key type=, default='event_description' output_key type=, default=None max_token_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None sub_doc_template type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: nlpaug_en_mapper 【算子类】: NlpaugEnMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Augments English text samples using various methods from the nlpaug library. 【支持参数列表（YAML可直接使用）】 sequential type=, default=False aug_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_original_sample type=, default=True delete_random_word type=, default=False swap_random_word type=, default=False spelling_error_word type=, default=False split_random_word type=, default=False keyboard_error_char type=, default=False ocr_error_char type=, default=False delete_random_char type=, default=False swap_random_char type=, default=False insert_random_char type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: nlpcda_zh_mapper 【算子类】: NlpcdaZhMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Augments Chinese text samples using the nlpcda library. 【支持参数列表（YAML可直接使用）】 sequential type=, default=False aug_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_original_sample type=, default=True replace_similar_word type=, default=False replace_homophone_char type=, default=False delete_random_char type=, default=False swap_random_char type=, default=False replace_equivalent_num type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: optimize_prompt_mapper 【算子类】: OptimizePromptMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Optimize prompts based on existing ones in the same batch. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='Qwen/Qwen2.5-7B-Instruct' gen_num type=typing.Annotated[int, Gt(gt=0)], default=3 max_example_num type=typing.Annotated[int, Gt(gt=0)], default=3 keep_original_sample type=, default=True retry_num type=, default=3 api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None example_template type=typing.Optional[str], default=None prompt_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None enable_vllm type=, default=False is_hf_model type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: optimize_qa_mapper 【算子类】: OptimizeQAMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Mapper to optimize question-answer pairs. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='Qwen/Qwen2.5-7B-Instruct' is_hf_model type=, default=True api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: optimize_query_mapper 【算子类】: OptimizeQueryMapper 【算子类型】: OptimizeQAMapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Optimize queries in question-answer pairs to make them more specific and detailed. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='Qwen/Qwen2.5-7B-Instruct' is_hf_model type=, default=True api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: optimize_response_mapper 【算子类】: OptimizeResponseMapper 【算子类型】: OptimizeQAMapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Optimize response in question-answer pairs to be more detailed and specific. 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='Qwen/Qwen2.5-7B-Instruct' is_hf_model type=, default=True api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None qa_pair_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 enable_vllm type=, default=False model_params type=typing.Optional[typing.Dict], default=None sampling_params type=typing.Optional[typing.Dict], default=None kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: pair_preference_mapper 【算子类】: PairPreferenceMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to construct paired preference samples by generating a rejected response and its 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern type=typing.Optional[str], default=None rejected_key type=, default='rejected_response' reason_key type=, default='reason' try_num type=typing.Annotated[int, Gt(gt=0)], default=3 model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: perplexity_filter 【算子类】: PerplexityFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with perplexity score in a specified range. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' min_ppl type=, default=0 max_ppl type=, default=1500 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: phrase_grounding_recall_filter 【算子类】: PhraseGroundingRecallFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples based on the phrase grounding recall of phrases extracted from 【支持参数列表（YAML可直接使用）】 hf_owlvit type=, default='google/owlvit-base-patch32' trust_remote_code type=, default=False min_recall type=, default=0.1 max_recall type=, default=1.0 horizontal_flip type=, default=False vertical_flip type=, default=False any_or_all type=, default='any' reduce_mode type=, default='avg' iou_thr type=, default=0.5 large_area_ratio_thr type=, default=0.95 conf_thr type=, default=0.0 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: punctuation_normalization_mapper 【算子类】: PunctuationNormalizationMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Normalizes unicode punctuations to their English equivalents in text samples. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: python_file_mapper 【算子类】: PythonFileMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Executes a Python function defined in a file on input data. 【支持参数列表（YAML可直接使用）】 file_path type=, default='' function_name type=, default='process_single' batched type=, default=False kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: python_lambda_mapper 【算子类】: PythonLambdaMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper for applying a Python lambda function to data samples. 【支持参数列表（YAML可直接使用）】 lambda_str type=, default='' batched type=, default=False kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: query_intent_detection_mapper 【算子类】: QueryIntentDetectionMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Predicts the user's intent label and corresponding score for a given query. The operator 【支持参数列表（YAML可直接使用）】 hf_model type=, default='bespin-global/klue-roberta-small-3i4k-intent-classification' zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en' model_params type=typing.Dict, default={} zh_to_en_model_params type=typing.Dict, default={} label_key type=, default='query_intent_label' score_key type=, default='query_intent_label_score' kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: query_sentiment_detection_mapper 【算子类】: QuerySentimentDetectionMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Predicts user's sentiment label ('negative', 'neutral', 'positive') in a query. 【支持参数列表（YAML可直接使用）】 hf_model type=, default='mrm8488/distilroberta-finetuned-financial-news-sentiment-analysis' zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en' model_params type=typing.Dict, default={} zh_to_en_model_params type=typing.Dict, default={} label_key type=, default='query_sentiment_label' score_key type=, default='query_sentiment_label_score' kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: query_topic_detection_mapper 【算子类】: QueryTopicDetectionMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Predicts the topic label and its corresponding score for a given query. The input is 【支持参数列表（YAML可直接使用）】 hf_model type=, default='dstefa/roberta-base_topic_classification_nyt_news' zh_to_en_hf_model type=typing.Optional[str], default='Helsinki-NLP/opus-mt-zh-en' model_params type=typing.Dict, default={} zh_to_en_model_params type=typing.Dict, default={} label_key type=, default='query_topic_label' score_key type=, default='query_topic_label_score' kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: random_selector 【算子类】: RandomSelector 【算子类型】: Selector 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Randomly selects a subset of samples from the dataset. 【支持参数列表（YAML可直接使用）】 select_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None select_num type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: range_specified_field_selector 【算子类】: RangeSpecifiedFieldSelector 【算子类型】: Selector 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Selects a range of samples based on the sorted values of a specified field. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' lower_percentile type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None upper_percentile type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None lower_rank type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None upper_rank type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: ray_bts_minhash_deduplicator 【算子类】: RayBTSMinhashDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: True 【文档说明】: A MinhashLSH deduplicator that operates in Ray distributed mode. 【支持参数列表（YAML可直接使用）】 tokenization type=, default='space' window_size type=typing.Annotated[int, Gt(gt=0)], default=5 lowercase type=, default=True ignore_pattern type=typing.Optional[str], default=None num_permutations type=typing.Annotated[int, Gt(gt=0)], default=256 jaccard_threshold type=typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])], default=0.7 num_bands type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None num_rows_per_band type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None tokenizer_model type=typing.Optional[str], default=None union_find_parallel_num type=typing.Union[int, str], default='auto' union_threshold type=typing.Optional[int], default=256 max_pending_edge_buffer_task type=typing.Optional[int], default=20 num_edge_buffer_task_returns type=typing.Optional[int], default=10 max_pending_filter_tasks type=typing.Optional[int], default=20 num_filter_task_returns type=typing.Optional[int], default=10 merge_batch_size type=typing.Optional[int], default=1000 minhash_batch_size type=typing.Union[str, int, NoneType], default='auto' memory_per_sample type=typing.Optional[float], default=0.1 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: ray_document_deduplicator 【算子类】: RayDocumentDeduplicator 【算子类型】: RayBasicDeduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using exact matching in Ray distributed mode. 【支持参数列表（YAML可直接使用）】 backend type=, default='ray_actor' redis_address type=, default='redis://localhost:6379' lowercase type=, default=False ignore_non_character type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: ray_image_deduplicator 【算子类】: RayImageDeduplicator 【算子类型】: RayBasicDeduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using exact matching of images in Ray distributed mode. 【支持参数列表（YAML可直接使用）】 backend type=, default='ray_actor' redis_address type=, default='redis://localhost:6379' method type=, default='phash' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: ray_video_deduplicator 【算子类】: RayVideoDeduplicator 【算子类型】: RayBasicDeduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at document-level using exact matching of videos in Ray distributed mode. 【支持参数列表（YAML可直接使用）】 backend type=, default='ray_actor' redis_address type=, default='redis://localhost:6379' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: relation_identity_mapper 【算子类】: RelationIdentityMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Identify the relation between two entities in a given text. 【支持参数列表（YAML可直接使用）】 api_model type=, default='gpt-4o' source_entity type=, default=None target_entity type=, default=None output_key type=, default='role_relation' api_endpoint type=typing.Optional[str], default=None response_path type=typing.Optional[str], default=None system_prompt_template type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None output_pattern_template type=typing.Optional[str], default=None try_num type=typing.Annotated[int, Gt(gt=0)], default=3 drop_text type=, default=False model_params type=typing.Dict, default={} sampling_params type=typing.Dict, default={} kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_bibliography_mapper 【算子类】: RemoveBibliographyMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Removes bibliography sections at the end of LaTeX documents. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_comments_mapper 【算子类】: RemoveCommentsMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Removes comments from documents, currently supporting only 'tex' format. 【支持参数列表（YAML可直接使用）】 doc_type type=typing.Union[str, typing.List[str]], default='tex' inline type=, default=True multiline type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_header_mapper 【算子类】: RemoveHeaderMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Removes headers at the beginning of documents in LaTeX samples. 【支持参数列表（YAML可直接使用）】 drop_no_head type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_long_words_mapper 【算子类】: RemoveLongWordsMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to remove long words within a specific range. 【支持参数列表（YAML可直接使用）】 min_len type=, default=1 max_len type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_non_chinese_character_mapper 【算子类】: RemoveNonChineseCharacterlMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Removes non-Chinese characters from text samples. 【支持参数列表（YAML可直接使用）】 keep_alphabet type=, default=True keep_number type=, default=True keep_punc type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_repeat_sentences_mapper 【算子类】: RemoveRepeatSentencesMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to remove repeat sentences in text samples. 【支持参数列表（YAML可直接使用）】 lowercase type=, default=False ignore_special_character type=, default=True min_repeat_sentence_length type=, default=2 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_specific_chars_mapper 【算子类】: RemoveSpecificCharsMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Removes specific characters from text samples. 【支持参数列表（YAML可直接使用）】 chars_to_remove type=typing.Union[str, typing.List[str]], default='◆●■►▼▲▴∆▻▷❖♡□' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_table_text_mapper 【算子类】: RemoveTableTextMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to remove table texts from text samples. 【支持参数列表（YAML可直接使用）】 min_col type=typing.Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=2), Le(le=20)])], default=2 max_col type=typing.Annotated[int, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=2), Le(le=20)])], default=20 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: remove_words_with_incorrect_substrings_mapper 【算子类】: RemoveWordsWithIncorrectSubstringsMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to remove words containing specified incorrect substrings. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' tokenization type=, default=False substrings type=typing.Optional[typing.List[str]], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: replace_content_mapper 【算子类】: ReplaceContentMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Replaces content in the text that matches a specific regular expression pattern with a 【支持参数列表（YAML可直接使用）】 pattern type=typing.Union[str, typing.List[str], NoneType], default=None repl type=typing.Union[str, typing.List[str]], default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: sdxl_prompt2prompt_mapper 【算子类】: SDXLPrompt2PromptMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates pairs of similar images using the SDXL model. 【支持参数列表（YAML可直接使用）】 hf_diffusion type=, default='stabilityai/stable-diffusion-xl-base-1.0' trust_remote_code type=, default=False torch_dtype type=, default='fp32' num_inference_steps type=, default=50 guidance_scale type=, default=7.5 text_key type=, default=None text_key_second type=, default=None output_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: sentence_augmentation_mapper 【算子类】: SentenceAugmentationMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Augments sentences by generating enhanced versions using a Hugging Face model. This 【支持参数列表（YAML可直接使用）】 hf_model type=, default='Qwen/Qwen2-7B-Instruct' system_prompt type=, default=None task_sentence type=, default=None max_new_tokens type=, default=256 temperature type=, default=0.2 top_p type=, default=None num_beams type=, default=1 text_key type=, default=None text_key_second type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: sentence_split_mapper 【算子类】: SentenceSplitMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Splits text samples into individual sentences based on the specified language. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: special_characters_filter 【算子类】: SpecialCharactersFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with special-character ratio within a specific range. 【支持参数列表（YAML可直接使用）】 min_ratio type=, default=0.0 max_ratio type=, default=0.25 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: specified_field_filter 【算子类】: SpecifiedFieldFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter samples based on the specified field information. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' target_value type=typing.List, default=[] args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: specified_numeric_field_filter 【算子类】: SpecifiedNumericFieldFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter samples based on a specified numeric field value. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' min_value type=, default=-9223372036854775807 max_value type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: stopwords_filter 【算子类】: StopWordsFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with stopword ratio within a specified range. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' tokenization type=, default=False min_ratio type=, default=0.3 max_ratio type=, default=1.0 stopwords_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' use_words_aug type=, default=False words_aug_group_sizes type=typing.List[typing.Annotated[int, Gt(gt=0)]], default=[2] words_aug_join_char type=, default='' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: suffix_filter 【算子类】: SuffixFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with specified suffix. 【支持参数列表（YAML可直接使用）】 suffixes type=typing.Union[str, typing.List[str]], default=[] args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: tags_specified_field_selector 【算子类】: TagsSpecifiedFieldSelector 【算子类型】: Selector 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Selector to filter samples based on the tags of a specified field. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' target_tags type=typing.List[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_action_filter 【算子类】: TextActionFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep texts that contain a minimum number of actions. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' min_action_num type=, default=1 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_chunk_mapper 【算子类】: TextChunkMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Split input text into chunks based on specified criteria. 【支持参数列表（YAML可直接使用）】 max_len type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None split_pattern type=typing.Optional[str], default='\\n\\n' overlap_len type=typing.Annotated[int, Ge(ge=0)], default=0 tokenizer type=typing.Optional[str], default=None trust_remote_code type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_embd_similarity_filter 【算子类】: TextEmbdSimilarityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep texts whose average embedding similarity to a set of given validation 【支持参数列表（YAML可直接使用）】 api_or_hf_model type=, default='text-embedding-v4' is_hf_model type=, default=False api_endpoint type=, default='embeddings' response_path type=, default='data.0.embedding' model_params type=typing.Optional[typing.Dict], default=None min_score type=, default=0.1 max_score type=, default=1.0 valid_dataset type=typing.Optional[typing.List[typing.Dict]], default=None ebd_dim type=, default=4096 pooling type=typing.Optional[str], default=None input_template type=typing.Optional[str], default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_entity_dependency_filter 【算子类】: TextEntityDependencyFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Identify and filter text samples based on entity dependencies. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' min_dependency_num type=, default=1 any_or_all type=, default='all' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_length_filter 【算子类】: TextLengthFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with total text length within a specific range. 【支持参数列表（YAML可直接使用）】 min_len type=, default=10 max_len type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: text_pair_similarity_filter 【算子类】: TextPairSimilarityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep text pairs with similarities within a specific range. 【支持参数列表（YAML可直接使用）】 hf_clip type=, default='openai/clip-vit-base-patch32' trust_remote_code type=, default=False min_score type=, default=0.1 max_score type=, default=1.0 text_key_second type=, default=None any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: token_num_filter 【算子类】: TokenNumFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with a total token number within a specified range. 【支持参数列表（YAML可直接使用）】 hf_tokenizer type=, default='EleutherAI/pythia-6.9b-deduped' min_num type=, default=10 max_num type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: topk_specified_field_selector 【算子类】: TopkSpecifiedFieldSelector 【算子类型】: Selector 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Selects top samples based on the sorted values of a specified field. 【支持参数列表（YAML可直接使用）】 field_key type=, default='' top_ratio type=typing.Optional[typing.Annotated[float, FieldInfo(annotation=NoneType, required=True, metadata=[Ge(ge=0), Le(le=1)])]], default=None topk type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None reverse type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: vggt_mapper 【算子类】: VggtMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Input a video of a single scene, and use VGGT to extract information including Camera 【支持参数列表（YAML可直接使用）】 vggt_model_path type=, default='facebook/VGGT-1B' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 duration type=, default=0 tag_field_name type=, default='vggt_tags' frame_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' if_output_camera_parameters type=, default=True if_output_depth_maps type=, default=True if_output_point_maps_from_projection type=, default=True if_output_point_maps_from_unprojection type=, default=True if_output_point_tracks type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_aesthetics_filter 【算子类】: VideoAestheticsFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep data samples with aesthetics scores for specified frames in the videos 【支持参数列表（YAML可直接使用）】 hf_scorer_model type=, default='' trust_remote_code type=, default=False min_score type=, default=0.4 max_score type=, default=1.0 frame_sampling_method type=, default='uniform' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 any_or_all type=, default='any' reduce_mode type=, default='avg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_aspect_ratio_filter 【算子类】: VideoAspectRatioFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with video aspect ratio within a specific range. 【支持参数列表（YAML可直接使用）】 min_ratio type=, default='9/21' max_ratio type=, default='21/9' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_captioning_from_audio_mapper 【算子类】: VideoCaptioningFromAudioMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Mapper to caption a video according to its audio streams based on 【支持参数列表（YAML可直接使用）】 keep_original_sample type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_captioning_from_frames_mapper 【算子类】: VideoCaptioningFromFramesMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Generates video captions from sampled frames using an image-to-text model. Captions from 【支持参数列表（YAML可直接使用）】 hf_img2seq type=, default='Salesforce/blip2-opt-2.7b' trust_remote_code type=, default=False caption_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_candidate_mode type=, default='random_any' keep_original_sample type=, default=True prompt type=typing.Optional[str], default=None prompt_key type=typing.Optional[str], default=None frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 horizontal_flip type=, default=False vertical_flip type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_captioning_from_summarizer_mapper 【算子类】: VideoCaptioningFromSummarizerMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Mapper to generate video captions by summarizing several kinds of generated 【支持参数列表（YAML可直接使用）】 hf_summarizer type=, default=None trust_remote_code type=, default=False consider_video_caption_from_video type=, default=True consider_video_caption_from_audio type=, default=True consider_video_caption_from_frames type=, default=True consider_video_tags_from_audio type=, default=True consider_video_tags_from_frames type=, default=True vid_cap_from_vid_args type=typing.Optional[typing.Dict], default=None vid_cap_from_frm_args type=typing.Optional[typing.Dict], default=None vid_tag_from_aud_args type=typing.Optional[typing.Dict], default=None vid_tag_from_frm_args type=typing.Optional[typing.Dict], default=None keep_tag_num type=typing.Annotated[int, Gt(gt=0)], default=5 keep_original_sample type=, default=True args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_captioning_from_video_mapper 【算子类】: VideoCaptioningFromVideoMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: True 【文档说明】: Generates video captions using a Hugging Face video-to-text model and sampled video 【支持参数列表（YAML可直接使用）】 hf_video_blip type=, default='kpyu/video-blip-opt-2.7b-ego4d' trust_remote_code type=, default=False caption_num type=typing.Annotated[int, Gt(gt=0)], default=1 keep_candidate_mode type=, default='random_any' keep_original_sample type=, default=True prompt type=typing.Optional[str], default=None prompt_key type=typing.Optional[str], default=None frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 horizontal_flip type=, default=False vertical_flip type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_deduplicator 【算子类】: VideoDeduplicator 【算子类型】: Deduplicator 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Deduplicates samples at the document level using exact matching of videos. 【支持参数列表（YAML可直接使用）】 consider_text type=, default=False args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_duration_filter 【算子类】: VideoDurationFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Keep data samples whose videos' durations are within a specified range. 【支持参数列表（YAML可直接使用）】 min_duration type=, default=0 max_duration type=, default=9223372036854775807 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_extract_frames_mapper 【算子类】: VideoExtractFramesMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Mapper to extract frames from video files according to specified methods. 【支持参数列表（YAML可直接使用）】 frame_sampling_method type=, default='all_keyframes' output_format type=, default='path' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 duration type=, default=0 frame_dir type=, default=None frame_key type=, default='video_frames' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_face_blur_mapper 【算子类】: VideoFaceBlurMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Mapper to blur faces detected in videos. 【支持参数列表（YAML可直接使用）】 cv_classifier type=, default='' blur_type type=, default='gaussian' radius type=, default=2 save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_ffmpeg_wrapped_mapper 【算子类】: VideoFFmpegWrappedMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Wraps FFmpeg video filters for processing video files in a dataset. 【支持参数列表（YAML可直接使用）】 filter_name type=typing.Optional[str], default=None filter_kwargs type=typing.Optional[typing.Dict], default=None global_args type=typing.Optional[typing.List[str]], default=None capture_stderr type=, default=True overwrite_output type=, default=True save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_frames_text_similarity_filter 【算子类】: VideoFramesTextSimilarityFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples based on the similarity between video frame images and text 【支持参数列表（YAML可直接使用）】 hf_clip type=, default='openai/clip-vit-base-patch32' trust_remote_code type=, default=False min_score type=, default=0.1 max_score type=, default=1.0 frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 horizontal_flip type=, default=False vertical_flip type=, default=False any_or_all type=, default='any' reduce_mode type=, default='avg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_hand_reconstruction_mapper 【算子类】: VideoHandReconstructionMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Use the WiLoR model for hand localization and 【支持参数列表（YAML可直接使用）】 wilor_model_path type=, default='wilor_final.ckpt' wilor_model_config type=, default='model_config.yaml' detector_model_path type=, default='detector.pt' mano_right_path type=, default='path_to_mano_right_pkl' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 duration type=, default=0 batch_size type=, default=16 tag_field_name type=, default='hand_reconstruction_tags' frame_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' if_save_visualization type=, default=True save_visualization_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' if_save_mesh type=, default=True save_mesh_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_motion_score_filter 【算子类】: VideoMotionScoreFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Filter to keep samples with video motion scores within a specific range. 【支持参数列表（YAML可直接使用）】 min_score type=, default=0.25 max_score type=, default=1.7976931348623157e+308 sampling_fps type=typing.Annotated[float, Gt(gt=0)], default=2 size type=typing.Union[typing.Annotated[int, Gt(gt=0)], typing.Tuple[typing.Annotated[int, Gt(gt=0)]], typing.Tuple[typing.Annotated[int, Gt(gt=0)], typing.Annotated[int, Gt(gt=0)]], NoneType], default=None max_size type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None divisible type=typing.Annotated[int, Gt(gt=0)], default=1 relative type=, default=False any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_motion_score_raft_filter 【算子类】: VideoMotionScoreRaftFilter 【算子类型】: VideoMotionScoreFilter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples with video motion scores within a specified range. 【支持参数列表（YAML可直接使用）】 min_score type=, default=1.0 max_score type=, default=1.7976931348623157e+308 sampling_fps type=typing.Annotated[float, Gt(gt=0)], default=2 size type=typing.Union[typing.Annotated[int, Gt(gt=0)], typing.Tuple[typing.Annotated[int, Gt(gt=0)]], typing.Tuple[typing.Annotated[int, Gt(gt=0)], typing.Annotated[int, Gt(gt=0)]], NoneType], default=None max_size type=typing.Optional[typing.Annotated[int, Gt(gt=0)]], default=None divisible type=typing.Annotated[int, Gt(gt=0)], default=8 relative type=, default=False any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_nsfw_filter 【算子类】: VideoNSFWFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples whose videos have nsfw scores in a specified range. 【支持参数列表（YAML可直接使用）】 hf_nsfw_model type=, default='Falconsai/nsfw_image_detection' trust_remote_code type=, default=False min_score type=, default=0.0 max_score type=, default=0.5 frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 reduce_mode type=, default='avg' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_ocr_area_ratio_filter 【算子类】: VideoOcrAreaRatioFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Keep data samples whose detected text area ratios for specified frames in the video are 【支持参数列表（YAML可直接使用）】 min_area_ratio type=, default=0 max_area_ratio type=, default=1.0 frame_sample_num type=typing.Annotated[int, Gt(gt=0)], default=3 languages_to_detect type=typing.Union[str, typing.List[str]], default=['ch_sim', 'en'] any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_remove_watermark_mapper 【算子类】: VideoRemoveWatermarkMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Remove watermarks from videos based on specified regions. 【支持参数列表（YAML可直接使用）】 roi_strings type=typing.List[str], default=['0,0,0.1,0.1'] roi_type type=, default='ratio' roi_key type=typing.Optional[str], default=None frame_num type=typing.Annotated[int, Gt(gt=0)], default=10 min_frame_threshold type=typing.Annotated[int, Gt(gt=0)], default=7 detection_method type=, default='pixel_value' save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_resize_aspect_ratio_mapper 【算子类】: VideoResizeAspectRatioMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Resizes videos to fit within a specified aspect ratio range. This operator adjusts the 【支持参数列表（YAML可直接使用）】 min_ratio type=, default='9/21' max_ratio type=, default='21/9' strategy type=, default='increase' save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_resize_resolution_mapper 【算子类】: VideoResizeResolutionMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Resizes video resolution based on specified width and height constraints. 【支持参数列表（YAML可直接使用）】 min_width type=, default=1 max_width type=, default=9223372036854775807 min_height type=, default=1 max_height type=, default=9223372036854775807 force_original_aspect_ratio type=, default='disable' force_divisible_by type=typing.Annotated[int, Gt(gt=0)], default=2 save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_resolution_filter 【算子类】: VideoResolutionFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Keep data samples whose videos' resolutions are within a specified range. 【支持参数列表（YAML可直接使用）】 min_width type=, default=1 max_width type=, default=9223372036854775807 min_height type=, default=1 max_height type=, default=9223372036854775807 any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_split_by_duration_mapper 【算子类】: VideoSplitByDurationMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Splits videos into segments based on a specified duration. 【支持参数列表（YAML可直接使用）】 split_duration type=, default=10 min_last_split_duration type=, default=0 keep_original_sample type=, default=True save_dir type=, default=None video_backend type=, default='ffmpeg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_split_by_key_frame_mapper 【算子类】: VideoSplitByKeyFrameMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Splits a video into segments based on key frames. 【支持参数列表（YAML可直接使用）】 keep_original_sample type=, default=True save_dir type=, default=None video_backend type=, default='ffmpeg' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_split_by_scene_mapper 【算子类】: VideoSplitBySceneMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: False 【文档说明】: Splits videos into scene clips based on detected scene changes. 【支持参数列表（YAML可直接使用）】 detector type=, default='ContentDetector' threshold type=typing.Annotated[float, Ge(ge=0)], default=27.0 min_scene_len type=typing.Annotated[int, Ge(ge=0)], default=15 show_progress type=, default=False save_dir type=, default=None args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_tagging_from_audio_mapper 【算子类】: VideoTaggingFromAudioMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates video tags from audio streams using the Audio Spectrogram Transformer. 【支持参数列表（YAML可直接使用）】 hf_ast type=, default='MIT/ast-finetuned-audioset-10-10-0.4593' trust_remote_code type=, default=False tag_field_name type=, default='video_audio_tags' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_tagging_from_frames_filter 【算子类】: VideoTaggingFromFramesFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples whose videos contain specified tags. 【支持参数列表（YAML可直接使用）】 tags type=typing.List[str], default=['people'] contain type=, default='any' frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 tag_field_name type=, default='video_frame_tags' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_tagging_from_frames_mapper 【算子类】: VideoTaggingFromFramesMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Generates video tags from frames extracted from videos. 【支持参数列表（YAML可直接使用）】 frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 tag_field_name type=, default='video_frame_tags' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_watermark_filter 【算子类】: VideoWatermarkFilter 【算子类型】: Filter 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Filter to keep samples whose videos have no watermark with high probability. 【支持参数列表（YAML可直接使用）】 hf_watermark_model type=, default='amrul-hzz/watermark_detector' trust_remote_code type=, default=False prob_threshold type=, default=0.8 frame_sampling_method type=, default='all_keyframes' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 reduce_mode type=, default='avg' any_or_all type=, default='any' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: video_whole_body_pose_estimation_mapper 【算子类】: VideoWholeBodyPoseEstimationMapper 【算子类型】: Mapper 【加速设备】: cuda 【批处理算子】: False 【文档说明】: Input a video containing people, and use the DWPose 【支持参数列表（YAML可直接使用）】 onnx_det_model type=, default='yolox_l.onnx' onnx_pose_model type=, default='dw-ll_ucoco_384.onnx' frame_num type=typing.Annotated[int, Gt(gt=0)], default=3 duration type=, default=0 tag_field_name type=, default='pose_estimation_tags' frame_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' if_save_visualization type=, default=False save_visualization_dir type=, default='C:\\Users\\Administrator/.cache\\data_juicer\\assets' args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: whitespace_normalization_mapper 【算子类】: WhitespaceNormalizationMapper 【算子类型】: Mapper 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Normalizes various types of whitespace characters to standard spaces in text samples. 【支持参数列表（YAML可直接使用）】 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: word_repetition_filter 【算子类】: WordRepetitionFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with word-level n-gram repetition ratio within a specific range. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' tokenization type=, default=False rep_len type=typing.Annotated[int, Gt(gt=0)], default=10 min_ratio type=, default=0.0 max_ratio type=, default=0.5 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ 【算子名称】: words_num_filter 【算子类】: WordsNumFilter 【算子类型】: Filter 【加速设备】: cpu 【批处理算子】: True 【文档说明】: Filter to keep samples with a total word count within a specified range. 【支持参数列表（YAML可直接使用）】 lang type=, default='en' tokenization type=, default=False min_num type=, default=10 max_num type=, default=9223372036854775807 args type=, default=required (无默认值) kwargs type=, default=required (无默认值) ================================================================================ (python3.11) D:\xu_projects\for_mage\llm\repos\DeepData\test\ray>