![]() ![]() Job attempt directory in $dest/_temporary/$jobAttemptId/ contains all output of the job in progress every task attempt is allocated its own task attempt dir $dest/_temporary/$jobAttemptId/_temporary/$taskAttemptIdĪll work for a task is written under the task attempt directory. File Output Committer V1 and V2 File Output Committer V1 and V2 Commit algorithms Task attempt execution (V1 and V2) It is possible for multiple task attempts to get their data into the output directory tree, and if a job fails/is aborted before the job is committed, thie output is visible. The v2 algorithm is not considered safe because the output is visible when individual tasks commit, rather than being delayed until job commit. The v1 algorithm is resilient to all forms of task failure, but slow when committing the final aggregate output as it renames each newly created file to the correct place in the table one by one. The committer built into hadoop-mapreduce-client-core module is the FileOutputCommitter.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |