An encoding system for a data set, particularly for a video data set is disclosed in the present invention. The encoding system includes a coding strategies represented as a tree-based structure, a cost projection system for each tree node, and a parameter searching algorithm for coding strategies to encode a data set. When the encoder parameters are obtained, the coding strategies and rate-allocation have better performance comparing to the related art.