CN102880603A - Method and equipment for filtering ranking list data - Google Patents

Abstract

The embodiment of the invention discloses a method for filtering ranking list data. The method comprises the following steps that: data filtering equipment acquires original data before a ranking date in a preset time period from data storage equipment; the data filtering equipment performs interpolation processing on the original data to obtain historical data according to the defect degree of the original data; the data filtering equipment calculates according to the historical data and the ranking date data and determines the list entrance probability of the ranking date data; and the data filtering equipment filters the ranking date data when the list entrance probability of the ranking date data is smaller than or equal to a probability threshold value. According to the technical scheme provided by the embodiment, a method for calculating the list entrance probability whether the ranking date data can enter the ranking list by taking the historical data as a basis and determining whether the ranking date data is filtered according to the list entrance probability is provided, and the ranking list data filtering accuracy is improved.

Description

A kind of method and apparatus of ranking list data filtering

Technical field

The application relates to field of computer technology, particularly a kind of method and apparatus of ranking list data filtering.

Background technology

Ranking list is goed deep into everybody life as a very concerned things of modern society.For example, the electronic business transaction website can provide the sales volume ranking list of various product to supply with consumer's (for convenience of description, with the consumer referred to as the buyer) or businessman (for convenience of description, with businessman referred to as the seller) reference.

In the prior art, ranking list is normally by sorting obtains to all raw data (for example Sales Volume of Commodity ranking list, its raw data is the sales volume of extensive stock).But the efficient that this mode causes calculating ranking list is very low, especially in the very large situation of original data volume, adopts this mode to be difficult to obtain rapidly ranking list.

In order to overcome the problems referred to above, a kind of ranking list data filtering method is provided in the prior art, the method is hereinafter referred to as the threshold filtering method.The threshold filtering method presets threshold value, only the data greater than threshold value is sorted, and filters out the following data of threshold value.This mode can reduce the data volume that participates in ordering, and still, there is irrational situation in the setting of threshold value, and the data filtering that originally should not be filtered is fallen.For example, for the sales volume ranking list, suppose that threshold value is 5, then sales volume should be fallen at the data filtering below 5 that still, the overall sales volume of some commodity itself is just very low, even this is just so that sales volume still may advance list at the commodity below 5.Therefore, existing threshold filtering method can't be filtered raw data accurately.

Summary of the invention

The embodiment of the present application provides a kind of method and apparatus of ranking list data filtering, the problem that the ranking list data filtering method that passes through simple setting sales volume threshold value that solution exists in the prior art can't filter raw data is accurately filtered raw data accurately.

For achieving the above object, the embodiment of the present application provides a kind of ranking list data filtering method on the one hand, comprising:

Data filtering equipment obtains the raw data in the Preset Time section before the seniority among brothers and sisters day from data storage device;

Data filtering equipment is processed described raw data value of benefit according to the defect degree of described raw data and is obtained historical data;

Data filtering equipment according to described historical data and described seniority among brothers and sisters day data calculate, that determines seniority among brothers and sisters day data advances the list probability;

Data filtering equipment filters described seniority among brothers and sisters day data when the advancing the list probability and be less than or equal to probability threshold value of seniority among brothers and sisters day data.

On the other hand, the embodiment of the present application also provides a kind of data filtering equipment, comprising:

Acquiring unit is used for obtaining the raw data in the Preset Time section before the seniority among brothers and sisters day from data storage device;

Processing unit obtains historical data for according to the defect degree of described raw data described raw data being processed;

Computing unit, be used for according to described historical data and described seniority among brothers and sisters day data calculate, that determines seniority among brothers and sisters day data advances the list probability;

Filter element is used for when the advancing the list probability and be less than or equal to probability threshold value of seniority among brothers and sisters day data, filters described seniority among brothers and sisters day data.

Compared with prior art, the embodiment of the present application has the following advantages:

Enter the comparison between the historical data of the probability of ranking list and seniority among brothers and sisters day data and objects of statistics according to historical data, that calculates seniority among brothers and sisters day data advances the list probability, if advance the list probability greater than probability threshold value, then do not filter seniority among brothers and sisters day data, if advance the list probability less than probability threshold value, then filter seniority among brothers and sisters day data.The ranking list data filtering method that provides by the embodiment of the present application, provide a kind of take historical data as advancing the list probability according to what calculate that can seniority among brothers and sisters day data enter ranking list, advance the list probability according to this and whether determine whether to filter seniority among brothers and sisters day data greater than the probability threshold value of a certain setting, improved the accuracy of ranking list data filtering.

Description of drawings

Fig. 1 is the schematic flow sheet of a kind of ranking list data filtering method of the embodiment of the present application proposition;

Fig. 2 is the schematic flow sheet of a kind of damaged data benefit value method of the embodiment of the present application proposition;

Fig. 3 is the structural representation of a kind of data filtering equipment of the embodiment of the present application proposition.

Embodiment

As stated in the Background Art, ranking list data filtering method of the prior art is by setting threshold, and filters numerical value and carry out the filtration of ranking list raw data less than the data of threshold value, and this method can't be filtered raw data accurately.

In order to address the deficiencies of the prior art, the application has proposed a kind of ranking list data filtering method.When the data of current participation seniority among brothers and sisters were filtered, the raw data before adopting in the certain hour section determined whether the data of current participation seniority among brothers and sisters need to filter as a reference.The value of this time period needs pre-configured, can obtain for empirical value or according to preset strategy.

Below as an example of the sales volume ranking list example the ranking list data filtering method that provides of explanation the embodiment of the present application, with commodity as objects of statistics, need explanation, the ranking list data filtering method that the application provides can also be applicable to the ranking list of other types, such as popularity ranking list etc., and objects of statistics corresponding to the ranking list that is applicable to other types, such as popularity index etc.

As shown in Figure 1, the method may further comprise the steps:

Step S101, data filtering equipment obtains raw data in the predefined time period from data storage device.

Data filtering equipment directly obtains the sales volume data of different types of commodity from the data storage device that stores the data that need filtration, data filtering equipment carries out record with the data that get access in the mode that is suitable for filtering, in the present embodiment this mode can but be not limited to matrix-style, for example can also be the array form.Send in the data acquisition request of data storage device at data filtering equipment, can carry commodity number and the time period of the data of acquisition request.

For convenience, below take the form of raw data as matrix form as example, as follows:

$T={(T_1,T_2,......,T_m)}^{\text{'}}=\left[\begin{array}{cccccc}{t}_{11}& {\mathrm{<figure-callout\; id="12"\; label="t"\; filenames="HSA00000534884600022.png"\; state="\{\{state\}\}">t</figure-callout>}}_{12}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1n}\\ t_{21}& t_{22}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2n}\\ .& .& & .& & .\\ .& .& & .& & .\\ .& .& & .& & .\\ t_{i1}& t_{i2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{ij}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{in}}\\ .& .& & .& & .\\ .& .& & .& & .\\ .& .& & .& & .\\ t_{m1}& t_{m2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mj}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mn}}\end{array}\right],$

Wherein, T represents total commodity data.T _iThe data that represent i commodity, and i=(1 ..., m), m is the commodity number in the raw data, the value of this commodity number is pre-configured, can obtain for empirical value or according to preset strategy.T _i=(t _I1, t _I2..., t _In), n is the time period.P _j=(t _1j, t _2j..., t _Nj) expression j the day of trade data, j=(1 ..., n).t _IjThe data that represent j the day of trade of i commodity, in the embodiment of the present application, t _IjThe sales volumes of commodity in certain day of trade.In actual applications, for example in the popularity ranking list, t _IjIt can be the evaluating deg of commodity.

The embodiment of the invention describes with the sales volume data instance of m kind commodity in n days, and the filter method of the embodiment of the invention also can be applicable to the filtration with other forms of 2-D data, and other data that can be transformed to 2-D data.

Step S102, data filtering equipment is judged the defect degree of raw data, whether needs carry out data filtering to determine raw data according to the defect degree of raw data, if do not need data filtering, finish; If need to carry out data filtering, execution in step S103;

Step S103 judges whether raw data needs to carry out damaged data benefit value, if need data benefit value, execution in step S104, otherwise, directly with raw data as historical data, execution in step S105.

The judgment mode shown in step S102 and S103 of enumerating except the embodiment of the invention, whether needs carry out damaged data benefit value also can to determine first raw data according to the defect degree of raw data, if need data benefit value, execution in step S104, otherwise, continue need to judge whether data filtering, if do not need data filtering, finish; If need to carry out data filtering, execution in step S105.

Raw data may exist damaged.For a kind of commodity, when commodity in some time not on frame the time, corresponding data can be damaged in this time, for example, when a commodity among the step S101 when b the day of trade is not on frame, data t then _AbDamaged.One damaged threshold value is set in the embodiment of the present application, weighs the data defect degree of commodity.When damaged data surpass damaged threshold value, raw data is not carried out damaged data benefit value, do not carry out data filtering yet.When damaged data do not surpass damaged threshold value, need to judge whether raw data is carried out damaged data benefit value.Damaged threshold value can be damaged data number percent, also can be concrete fate, for example when the time period is 10 days, can set damaged threshold value is 5 days, if data are imperfect, and damaged data were above 5 days, then do not carry out damaged data benefit value, do not carry out data filtering yet.

Judge whether raw data needs to carry out damaged data benefit value and specifically comprise:

If the data within the time period are complete, then do not need to carry out damaged data benefit value; If the data within the time period are imperfect, and damaged data are lower than damaged threshold value, then carry out damaged data benefit value.

Carrying out respectively above-mentioned damaged benefit value for each commodity in the embodiment of the present application processes.

Step S104, the damaged data of data filtering equipment utilization entropy rationale Raw Data Simulation obtain damaged value complement with simulation and are charged into raw data, obtain historical data.

Damaged data produce serious influence to the data filtering in later stage.Consider that commodity do not represent sales volume on frame be 0, therefore with directly larger on the accuracy impact of rank results with the damaged data of 0 expression, a kind of correct thinking should be to simulate these commodity in the day trade transaction amount of frame.In information theory, information entropy is the tolerance that an information source is sent the contained quantity of information of a certain message, and more definite when the message that a certain information source is sent, the information entropy of this information source is just less.It is the tolerance of the unordered degree of system or confusion degree, has represented the average uncertainty of system.And Information Entropy is a kind of a kind of method of determining weight coefficient by the size of information that attribute value provides.It is strong that it has objectivity, the evaluation procedure transparency and the good characteristics of reproducibility.For example, for the attribute j that determines, the difference between j attribute of each data is larger, illustrates that then the relativity of this index is just larger, and namely its quantity of information is just larger, and entropy is less.

According to the entropy theory damaged raw data is simulated in the embodiment of the present application.

As shown in Figure 2, suppose the data t of b the day of trade of a commodity in the raw data _AbDamaged, with simulation t _AbDamaged value is that example is described as follows:

Step S1041, according to the non-damaged data of a commodity in the raw data, the entropy I of each day of trade that computational data is not damaged.

Suppose that arbitrary day of trade is f, the value of f is the not damaged days of trade of data, and then the entropy of f the day of trade obtains by following formula:

I _f＝-k·p _f·ln(p _f)

Wherein, k=1/ln (n), $p_f=\frac{d_f}{\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{d}_i},$

Wherein,

Represent f the day of trade data with damaged b the day of trade data distance, embodied both correlativity.t _AbDamaged, so i is not equal to a when calculating.

Step S1042 according to the entropy of f the day of trade, calculates the coefficient of variation of f the day of trade.

The coefficient of variation r of f the day of trade _f=1-I _f, f=1,2 ..., n,

Coefficient of variation is the amount of response data effect size, and its value is larger, and the effect of the data volume of f the day of trade is larger, and vice versa.

Step S1043 according to the coefficient of variation of f the day of trade, calculates the weight coefficient of f the day of trade.

$w_f=\frac{r_f}{\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{r}_i},f=\mathrm{1,2},...,n$

Step S1044 according to each data that calculate weight coefficient of the not damaged day of trade, calculates the damaged value t of b the day of trade _Ab

t _ab＝w ₁t _a1+w ₂t _a2+...w _(b-1)t _a(b-1)+w _(b+1)t _a(b+1)+...+w _nt _an

Fill up damaged value t thereby reach _AbEffect.

Above-mentionedly with a damaged data instance process of data benefit value has been described, marquis when the damaged a plurality of data of raw data and need to carry out data benefit value the time calculates respectively each damaged data according to noncolobomatous data in the raw data.

In the damaged value complement value method that the embodiment of the present application provides, considered in the raw data laterally and data dependence longitudinally that have good objectivity, the damaged value complementing method with respect to other has lower time complexity.

Step S105, according to historical data and seniority among brothers and sisters day data, that calculates commodity seniority among brothers and sisters day data advances the list probability.

Data filtering equipment replenishes the seniority among brothers and sisters data on day same day in historical data, i.e. seniority among brothers and sisters day data.

In the embodiment of the present application, that utilizes that Bayes (Bayes) model calculates seniority among brothers and sisters day data advances the list probability, and can its core be exactly to judge according to the sales volume on commodity seniority among brothers and sisters day same day that these commodity are current advance list under this sales volume, if can not then filter; Otherwise, then do not filter.That is, calculate can these commodity advance list the same day under this sales volume probability, if probability less than a certain specified value, then filters; Otherwise, then do not filter.In addition, advance the list probability and be one to be normalized to [0,1] interval interior value, the result that the model-naive Bayesian that provides by the embodiment of the present application draws, do not need to consider different merchandise classifications, the thought that does not namely adopt first classification to refilter is because can be by the historical data and seniority among brothers and sisters day data of these commodity of contrast in this method, thereby know that current seniority among brothers and sisters day data advance the probability of list, and the classificating thought of commodity is contained by this Idea of Probability in fact.

Concrete, step S105 comprises:

Step S1051 calculates the commodity historical data and enters the probability of ranking list, and the commodity historical data enters the probability of ranking list during less than seniority among brothers and sisters day data, and the commodity historical data is less than the probability of seniority among brothers and sisters day data.

When being the commodity historical data less than seniority among brothers and sisters day data, the probability that the commodity historical data enters ranking list during less than seniority among brothers and sisters day data enters the number of times of ranking list and the ratio of the number of times that commodity data enters ranking list, the probability that the commodity historical data enters ranking list is that commodity data enters the number of times of ranking list and the ratio of total degree in the time period, the number of times when the commodity historical data is the commodity historical data less than seniority among brothers and sisters day data less than the probability of seniority among brothers and sisters day data and the ratio of total degree in the time period.In the present embodiment, therefore ordering in a day once can represent that number of times calculates with fate.

Step S1052 enters the probability of ranking list according to the commodity historical data, and the commodity historical data enters the probability of ranking list during less than seniority among brothers and sisters day data, and the time period calculate commodity seniority among brothers and sisters day data advance the list probability.

The below describes the process of advancing the list probability of calculating commodity seniority among brothers and sisters day data among the step S105 with concrete example, can obtain partial data with seniority among brothers and sisters day data according to historical data to be:

$X={(X_1,X_2,......,X_m)}^{\text{'}}=\left[\begin{array}{ccccccc}{t}_{11}& {\mathrm{<figure-callout\; id="12"\; label="t"\; filenames="HSA00000534884600022.png"\; state="\{\{state\}\}">t</figure-callout>}}_{12}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1n}& x_{11}\\ t_{21}& t_{22}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2n}& x_{21}\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ t_{i1}& t_{i2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{ij}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{in}}& x_{i1}\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ t_{m1}& t_{m2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mj}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mn}}& x_{m1}\end{array}\right]$

Wherein, x=(x ₁₁, x ₂₁..., x _M1) ', being the seniority among brothers and sisters seniority among brothers and sisters day data on day same day, these data can be the trading volumes of commodity on the same day.Take i kind commodity as example, i kind commodity are X in the trading volume on the seniority among brothers and sisters day same day _I1, the historical data of commodity is (t _I1, t _I2..., t _In).U is for advancing the list data set in the order set, and V is not for advancing the list data set.Advance the list data set and do not advance the list data set and can carry out record by data server, when data filtering equipment obtains raw data from data server, can obtain simultaneously into the list data set and do not advance the list data set.

The probability that enters ranking list when (1), the commodity historical data is less than seniority among brothers and sisters day data is P (B|A):

$=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}(t_{\mathrm{ij}}<x_{i1})}{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)},(t_{\mathrm{ij}}\&Element;U)$

(2), to enter the probability of ranking list be P (A) to the commodity historical data:

$=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}{n},(t_{\mathrm{ij}}\&Element;U)$

(3), the commodity historical data is less than the probability P (B) of seniority among brothers and sisters day data:

$=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}({t_{\mathrm{ij}}}^{\&prime;}<x_{i1})}{n},({t_{\mathrm{ij}}}^{\&prime;}\&Element;(U\&cup;V))$

Wherein, the calculating in (1) (2) (3) is regardless of the front and back order.

(4), the list probability that advances of commodity seniority among brothers and sisters day data is P (A|B):

According to the result of calculation of (1) (2) (3), can obtain commodity the seniority among brothers and sisters day of seniority among brothers and sisters day data advance the list probability:

$P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)}$

$P\left(A\right|B=x_{i1})=\frac{\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}(t_{\mathrm{ij}}<x_{i1})}{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}\&CenterDot;\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}{n}}{\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}({t_{\mathrm{ij}}}^{\&prime;}<x_{i1})}{n}}$

Wherein, (t _Ij∈ U, t _Ij' ∈ (U ∪ V)).

For step S105, when calculating, that can calculate respectively each commodity seniority among brothers and sisters day data advances the list probability, in the present embodiment, when calculating a kind of commodity, only need the historical data of these commodity to get final product, that can utilize also that data in the matrix calculate all commodity seniority among brothers and sisters day data simultaneously advances the list probability.

Step S106 compares advancing list probability and probability threshold value, judges whether to filter the seniority among brothers and sisters day data of commodity according to comparative result.Concrete, if advance the list probability greater than probability threshold value, then do not filter the seniority among brothers and sisters day data of commodity, if advance the list probability less than probability threshold value, then filter the seniority among brothers and sisters day data of commodity.

Before step S106, comprise, set probability threshold value.

Wherein, probability threshold value is one and is normalized to [0,1] interval interior value that probability threshold value is the empirical value that can obtain by the data results of reality.

Need explanation, the method for utilizing the entropy theory to carry out damaged data benefit value among the step S104 is preferred benefit value mode, also can mend the value-based algorithm value of benefit according to other, for example mends value-based algorithm based on the damaged data of broad sense mahalanobis distance.

In the embodiment of the present application, enter the probability of ranking list and the comparison between seniority among brothers and sisters day data and the commodity historical data according to the commodity historical data, that calculates commodity seniority among brothers and sisters day data advances the list probability, if advance the list probability greater than probability threshold value, then do not filter the seniority among brothers and sisters day data of commodity, if advance the list probability less than probability threshold value, then filter the seniority among brothers and sisters day data of commodity.The ranking list data filtering method that provides by the embodiment of the present application, provide a kind of take historical data as advancing the list probability according to what calculate that can seniority among brothers and sisters day data enter ranking list, advance the list probability according to this and whether determine whether to filter seniority among brothers and sisters day data greater than the probability threshold value of a certain setting, improved the accuracy of ranking list data filtering.

In the ranking list data filtering method that above-described embodiment provides, considered the relation between the same commodity difference day of trade data, drawn comparatively accurately filter result.But, in the same day of trade, also exist between the data of different commodity and influence each other, further embodiment of this invention provides a kind of ranking list data filtering method, take into account also existing to influence each other between the data of different commodity of the same day of trade, that is, the longitudinal data correlation information in the above-mentioned partial data is also taken into account.

In the present embodiment, step S201～step S204 is with step S101～step S104.

Step S205, according to partial data, calculate commodity seniority among brothers and sisters day data advance the list probability.

In the present embodiment, therefore one day seniority among brothers and sisters once represents that with fate number of times calculates.

Concrete, step S205 comprises:

Step S2051, the probability of valid data difference when the calculating commodity enter ranking list, the probability of the data difference when the calculating commodity enter ranking list calculates commodity data difference general probability.

Wherein, the data of last day are seniority among brothers and sisters day data in the partial data, and other data before the seniority among brothers and sisters day data are historical data.The probability of valid data difference was the cumulative ratio of data difference in valid data difference in the fate of commodity when entering ranking list the cumulative fate when entering ranking list with commodity when commodity entered ranking list, the probability of the data difference when commodity enter ranking list is the cumulative ratio of cumulative and commodity data difference within the time period of data difference in the fate of commodity when entering ranking list, and commodity data difference general probability is the cumulative ratio of cumulative and commodity data difference within the time period of commodity valid data difference within the time period.

Step S2052, the probability of valid data difference when entering ranking list according to commodity, the probability of the data difference when commodity enter ranking list, and commodity data difference general probability, calculate commodity seniority among brothers and sisters day data advance the list probability.

The below describes calculating the process of advancing list probability of commodity in the day of trade among the step S205 with concrete example, and partial data is:

$X={(X_1,X_2,......,X_m)}^{\text{'}}=\left[\begin{array}{ccccccc}{t}_{11}& {\mathrm{<figure-callout\; id="12"\; label="t"\; filenames="HSA00000534884600022.png"\; state="\{\{state\}\}">t</figure-callout>}}_{12}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{1n}& x_{11}\\ t_{21}& t_{22}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2j}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{2n}& x_{21}\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ t_{i1}& t_{i2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{ij}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{in}}& x_{i1}\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ .& .& & .& & .& .\\ t_{m1}& t_{m2}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mj}}& \&CenterDot;\&CenterDot;\&CenterDot;& t_{\mathrm{mn}}& x_{m1}\end{array}\right]$

Wherein, x=(x ₁₁, x ₂₁..., x _M1) ', be the seniority among brothers and sisters data on day same day.

In the data filtering method that present embodiment provides, introduced the concept of data indexes, that be used for to calculate seniority among brothers and sisters day data advances the list probability.With r _IjRepresent i commodity at the data index of j the day of trade, in the present embodiment, r _IjBe the sales volume index, get the data of the part commodity of j the day of trade and calculate r as data sample _Ij, use r _IjRepresent that i commodity are in the sales volume status of j the day of trade in this data sample sales situation.

For certain data t _IjCorresponding r _Ij, r _Ij=rank (t _Ij), (j ∈ S _Ij), S _IjBe data sample, can get t _IjContiguous N _sIndividual data are as data sample S _Ij, i.e. r _IjExpression t _IjAt the contiguous N of j row _sThe rank of individual data, same, r _iBe seniority among brothers and sisters day data x _I1N in the vicinity of ranking day same day _sRank in the individual data, r _i=rank (x), (j ∈ S _i).

The big or small N of data sample _sCan be before calculating rule of thumb value set, for example, calculating r _IjThe time, the sample size that fetches data is 30, then gets t _Ij30 contiguous data are calculated, and in addition, can get j the day of trade with t _IjCentered by N _sIndividual data also can be got from t as data sample _IjThe N of beginning _sIndividual data are as data sample, and namely these 30 data can be (t _{(i-14) j}, t _{(i-13) j}..., t _{(i+15) j}) ', these 30 data also can be (t _Ij, t _{(i+1) j}..., t _{(i+29) j}) '.

Setting by the data index, be equivalent to certain day of trade data the impact of seniority among brothers and sisters day data has been increased a weight factor, when rationally controlling calculated amount, the whole transaction situation in the market on certain same day day of trade has been included in the scope of the consideration of calculating, weighed more accurately the impact of historical data on seniority among brothers and sisters day data, improved estimation and ranked the accuracy that day data are advanced the list probability.

The below specifies the process of advancing the list probability of calculating commodity seniority among brothers and sisters day data:

The probability of valid data difference was P (B|A) when (1), historical data entered ranking list:

$=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}},(t_{\mathrm{ij}}\&Element;U)$

The probability of the data difference when (2), historical data enters ranking list is P (A):

$=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))

(3), the data difference general probability is P (B):

$=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))

Wherein, the calculating in (1) (2) (3) is regardless of the front and back order.

(4), commodity seniority among brothers and sisters day data the list probability that advances be P (A|B):

According to the result of calculation of (1) (2) (3), can obtain commodity seniority among brothers and sisters day data advance the list probability:

$P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)}$

$P\left(A\right|B=x_{i1})=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}$

Wherein, (t _Ij∈ U, t _Ij' ∈ (U ∪ V))

Step S206 is with step S106.

In the embodiment of the present application, enter the probability of ranking list and the comparison between seniority among brothers and sisters day data and the commodity historical data according to the commodity historical data, that calculates commodity seniority among brothers and sisters day data advances the list probability, if advance the list probability greater than probability threshold value, then do not filter the seniority among brothers and sisters day data of commodity, if advance the list probability less than probability threshold value, then filter the seniority among brothers and sisters day data of commodity.The ranking list data filtering method that provides by the embodiment of the present application, provide a kind of take historical data as advancing the list probability according to what calculate that can seniority among brothers and sisters day data enter ranking list, advance the list probability according to this and whether determine whether to filter seniority among brothers and sisters day data greater than the probability threshold value of a certain setting, improved the accuracy of ranking list data filtering.

In order to realize the technical scheme of the embodiment of the present application, based on the technical conceive identical with said method embodiment, the embodiment of the present application also provides a kind of data filtering equipment, and its structural representation specifically comprises as shown in Figure 3:

Acquiring unit 11 is used for obtaining the raw data in the Preset Time section before the seniority among brothers and sisters day from data storage device;

Processing unit 12, the defect degree that is used for the raw data obtained according to described acquiring unit 11 is processed described raw data and is obtained historical data;

Computing unit 13 is used for calculating according to described historical data and seniority among brothers and sisters day data, and that determines seniority among brothers and sisters day data advances the list probability;

Filter element 14 was used for when described computing unit 13 calculates seniority among brothers and sisters day during the advancing the list probability and be less than or equal to probability threshold value of data, filtered described seniority among brothers and sisters day data.

Wherein, described computing unit 13 specifically is used for

Enter the probability of ranking list when it is calculated that the probability that obtains historical data and enter ranking list, historical data value less than seniority among brothers and sisters day data value according to described historical data and seniority among brothers and sisters number of days, and historical data is less than the probability of seniority among brothers and sisters day data; The probability, historical data value that enters ranking list according to described historical data enters the probability of ranking list and historical data during less than seniority among brothers and sisters day data value less than the probability of seniority among brothers and sisters day data, and that calculates seniority among brothers and sisters day data advances the list probability.

Enter the probability of ranking list when wherein, described historical data value is less than seniority among brothers and sisters day data value

$P\left(B\right|A)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}(t_{\mathrm{ij}}<x_{i1})}{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)},(t_{\mathrm{ij}}\&Element;U)$

Wherein, x _I1Be the seniority among brothers and sisters day data of i kind commodity, t _IjBe i commodity in the data of j the day of trade, U be historical data enter ranking list advance the list data set, V be historical data do not advance the list data set, n is the default time period;

Described historical data enters the probability of ranking list

$P\left(A\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}{n},(t_{\mathrm{ij}}\&Element;U)$

Described historical data is less than the probability of seniority among brothers and sisters day data

$P\left(B\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}({t_{\mathrm{ij}}}^{\&prime;}<x_{i1})}{n},(t_{\mathrm{ij}}\&Element;U,{t_{\mathrm{ij}}}^{\&prime;}\&Element;(U\&cup;V))$

Described seniority among brothers and sisters day data advance the list probability $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)}.$

Wherein, described computing unit 13 specifically is used for,

The probability of valid data difference when it is calculated that with the seniority among brothers and sisters number of days obtaining historical data enters ranking list according to described historical data, the probability of the data difference when historical data enters ranking list, and data difference general probability; The probability of valid data difference when entering ranking list according to described historical data, the probability of the data difference when historical data enters ranking list, and data difference general probability, that determines seniority among brothers and sisters day data advances the list probability.

The probability of valid data difference when wherein, described historical data enters ranking list

$P\left(B\right|A)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}},(t_{\mathrm{ij}}\&Element;U);$

Wherein, x _I1Be the data of described i kind commodity on same day seniority among brothers and sisters day, t _IjBe i commodity in the data of j the day of trade, r _IjBe t _IjRank in the data sample that the part commodity data forms in the j row, r _iBe x _I1Rank in the data sample that seniority among brothers and sisters part commodity data on day same day forms, U be historical data enter ranking list advance the list data set, n is the time period;

The probability of the data difference when described historical data enters ranking list

$P\left(A\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

Wherein, V be historical data do not advance the list data set;

Described data difference general probability

$P\left(B\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

Described seniority among brothers and sisters day data advance the list probability $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)}.$

Wherein, described processing unit 12 specifically is used for,

According to the non-damaged data in the raw data in the time period, calculate the entropy of each noncolobomatous day of trade of data; According to the entropy of the described day of trade, calculate the coefficient of variation in the described day of trade; According to the coefficient of variation of the described day of trade, calculate the weight coefficient of the described day of trade;

Weight coefficient according to each the noncolobomatous day of trade of data that calculates calculates damaged value.

Wherein, the damaged value of described calculating comprises:

The entropy I of f the day of trade _f=-kp _fLn (p _f);

Wherein, k=1/ln (n),

N participates in the time period of the time span of ordering for expression;

Wherein,

d _fBe used for f the day of trade of expression data and the distance between the damaged data; t _IfFor the value f the day of trade i kind commodity data, t _IbFor b the day of trade i kind commodity data;

The described coefficient of variation that calculates in the described day of trade according to the entropy of the described day of trade comprises:

r _f=1-I _f,, f=1 wherein, 2 ..., n;

The weight coefficient that described coefficient of variation according to the described day of trade calculates the described day of trade comprises:

$w_f=\frac{r_f}{\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{r}_i},f=\mathrm{1,2},...,m$

The weight coefficient of each noncolobomatous day of trade of data that described basis calculates calculates damaged value and comprises:

t _ab＝w ₁t _a1+w ₂t _a2+...w _(b-1)t _a(b-1)+w _(b+1)t _a(b+1)+...+w _nt _an

Wherein, t _AbBe damaged value.

The data filtering equipment that provides in the embodiment of the present application, enter the probability of ranking list and the comparison between seniority among brothers and sisters day data and the commodity historical data according to the commodity historical data, that calculates commodity seniority among brothers and sisters day data advances the list probability, if advance the list probability greater than probability threshold value, then do not filter the seniority among brothers and sisters day data of commodity, if advance the list probability less than probability threshold value, then filter the seniority among brothers and sisters day data of commodity.The ranking list data filtering method that provides by the embodiment of the present application, provide a kind of take historical data as advancing the list probability according to what calculate that can seniority among brothers and sisters day data enter ranking list, advance the list probability according to this and whether determine whether to filter seniority among brothers and sisters day data greater than the probability threshold value of a certain setting, improved the accuracy of ranking list data filtering.

Through the above description of the embodiments, those skilled in the art can be well understood to the embodiment of the present application and can realize by hardware, also can realize by the mode that software adds necessary general hardware platform.Based on such understanding, the technical scheme of the embodiment of the present application can embody with the form of software product, it (can be CD-ROM that this software product can be stored in a non-volatile memory medium, USB flash disk, portable hard drive etc.) in, comprise some instructions with so that computer equipment (can be personal computer, server, or the network equipment etc.) each implements the described method of scene to carry out the embodiment of the present application.

It will be appreciated by those skilled in the art that accompanying drawing is a preferred synoptic diagram of implementing scene, the module in the accompanying drawing or flow process might not be that enforcement the embodiment of the present application is necessary.

It will be appreciated by those skilled in the art that the module in the device of implementing in the scene can be distributed in the device of implementing scene according to implementing scene description, also can carry out respective change and be arranged in the one or more devices that are different from this enforcement scene.The module of above-mentioned enforcement scene can be merged into a module, also can further split into a plurality of submodules.

Above-mentioned the embodiment of the present application sequence number does not represent the quality of implementing scene just to description.

More than disclosed only be several implementation scenes of the embodiment of the present application, still, the embodiment of the present application is not limited thereto, the changes that any person skilled in the art can think of all should fall into the traffic limits scope of the embodiment of the present application.

Claims (14)

1. a ranking list data filtering method is characterized in that, comprising:

Data filtering equipment obtains the raw data in the Preset Time section before the seniority among brothers and sisters day from data storage device;

Data filtering equipment is processed described raw data value of benefit according to the defect degree of described raw data and is obtained historical data;

Data filtering equipment according to described historical data and described seniority among brothers and sisters day data calculate, that determines seniority among brothers and sisters day data advances the list probability;

Data filtering equipment filters described seniority among brothers and sisters day data when the advancing the list probability and be less than or equal to probability threshold value of seniority among brothers and sisters day data.

2. the method for claim 1 is characterized in that, described according to described historical data and described seniority among brothers and sisters day data calculate, determine that the list probability that advances of seniority among brothers and sisters day data comprises:

Enter the probability of ranking list and historical data value when it is calculated that the probability that obtains historical data and enter ranking list, historical data value less than seniority among brothers and sisters day data value according to described historical data and seniority among brothers and sisters number of days less than the probability of seniority among brothers and sisters day data value; The probability, historical data value that enters ranking list according to described historical data enters the probability of ranking list and historical data value during less than seniority among brothers and sisters day data value less than the probability of seniority among brothers and sisters day data value, and that calculates seniority among brothers and sisters day data advances the list probability.

3. method as claimed in claim 2 is characterized in that, the probability that described historical data value enters ranking list during less than seniority among brothers and sisters day data value obtains by following manner:

$P\left(B\right|A)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}(t_{\mathrm{ij}}<x_{i1})}{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)},(t_{\mathrm{ij}}\&Element;U)$

The probability that described historical data enters ranking list obtains by following manner:

$P\left(A\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}{n},(t_{\mathrm{ij}}\&Element;U)$

Described historical data obtains by following manner less than the probability of seniority among brothers and sisters day data:

$P\left(B\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}({t_{\mathrm{ij}}}^{\&prime;}<x_{i1})}{n},(t_{\mathrm{ij}}\&Element;U,{t_{\mathrm{ij}}}^{\&prime;}\&Element;(U\&cup;V))$

The list probability that advances of described seniority among brothers and sisters day data obtains by following manner: $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)};$

Wherein, x _I1Be the seniority among brothers and sisters day data of i kind objects of statistics, t _IjBe i kind objects of statistics in the data of j the day of trade, U be historical data enter ranking list advance the list data set, V be historical data do not advance the list data set, n is the default time period.

4. the method for claim 1 is characterized in that, described according to described historical data and described seniority among brothers and sisters day data calculate, determine that the list probability that advances of seniority among brothers and sisters day data comprises:

Probability and the data difference general probability of the probability of valid data difference, the data difference when historical data enters ranking list when it is calculated that with the seniority among brothers and sisters number of days obtaining historical data enters ranking list according to described historical data; The probability of the probability of valid data difference, the data difference when historical data enters ranking list and data difference general probability when entering ranking list according to described historical data, that determines seniority among brothers and sisters day data advances the list probability.

5. method as claimed in claim 4 is characterized in that, the probability of valid data difference obtained by following manner when described historical data entered ranking list:

$P\left(B\right|A)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}\&le;t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}},(t_{\mathrm{ij}}\&Element;U);$

The probability of the data difference when described historical data enters ranking list obtains by following manner:

$P\left(A\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

Described data difference general probability obtains by following manner:

$P\left(B\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

The list probability that advances of described seniority among brothers and sisters day data obtains by following manner: $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)};$

Wherein, x _I1Be the data of described i kind objects of statistics on same day seniority among brothers and sisters day, t _IjBe i kind objects of statistics in the data of j the day of trade, r _IjBe t _IjRank in the data sample that the data of part objects of statistics form in the j row, r _iBe x _I1Rank in the data sample that the data of seniority among brothers and sisters part objects of statistics on day same day form, U be historical data enter ranking list advance the list data set, V be historical data do not advance the list data set; N is the time period.

6. the method for claim 1 is characterized in that, described defect degree according to described raw data obtains historical data to the described raw data value of benefit processing and comprises:

Calculate the entropy of each noncolobomatous day of trade of data according to the non-damaged data in the raw data in the time period; Calculate coefficient of variation in the described day of trade according to the entropy of the described day of trade; Calculate the weight coefficient of the described day of trade according to the coefficient of variation of the described day of trade;

Weight coefficient according to each the noncolobomatous day of trade of data that calculates calculates damaged value.

7. method as claimed in claim 6 is characterized in that, the damaged value of described calculating comprises:

Calculate the entropy I of f the day of trade _f=-kp _fLn (p _f);

Wherein, k=1/ln (n), N participates in the time period of the time span of ordering for expression;

Wherein, d _fBe used for f the day of trade of expression data and the distance between the damaged data; t _IfFor f the day of trade i objects of statistics data, t _IbFor b the day of trade i kind objects of statistics data;

The described coefficient of variation that calculates in the described day of trade according to the entropy of the described day of trade comprises:

r _f=1-I _f,, f=1 wherein, 2 ..., n;

The weight coefficient that described coefficient of variation according to the described day of trade calculates the described day of trade comprises:

$w_f=\frac{r_f}{\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}{r}_i},f=\mathrm{1,2},...,m$

The weight coefficient of each noncolobomatous day of trade of data that described basis calculates calculates damaged value and comprises:

t _ab＝w ₁t _a1+w ₂t _a2+...w _(b-1)t _a(b-1)+w _(b+1)t _a(b+1)+...+w _nt _an

Wherein, t _AbBe damaged value.

8. a data filtering equipment is characterized in that, comprising:

Acquiring unit is used for obtaining the raw data in the Preset Time section before the seniority among brothers and sisters day from data storage device;

Processing unit is used for according to the defect degree of described raw data the described raw data value of benefit processing being obtained historical data;

Computing unit, be used for according to described historical data and described seniority among brothers and sisters day data calculate, that determines seniority among brothers and sisters day data advances the list probability;

Filter element is used for when the advancing the list probability and be less than or equal to probability threshold value of seniority among brothers and sisters day data, filters described seniority among brothers and sisters day data.

9. equipment as claimed in claim 8 is characterized in that, described computing unit specifically is used for:

Enter the probability of ranking list and historical data value when it is calculated that the probability that obtains historical data and enter ranking list, historical data value less than seniority among brothers and sisters day data value according to described historical data and seniority among brothers and sisters number of days less than the probability of seniority among brothers and sisters day data value; The probability, historical data value that enters ranking list according to described historical data enters the probability of ranking list and historical data value during less than seniority among brothers and sisters day data value less than the probability of seniority among brothers and sisters day data value, and that calculates seniority among brothers and sisters day data advances the list probability.

10. equipment as claimed in claim 9 is characterized in that, the probability that described historical data value enters ranking list during less than seniority among brothers and sisters day data value obtains by following manner:

$P\left(B\right|A)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}(t_{\mathrm{ij}}<x_{i1})}{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)},(t_{\mathrm{ij}}\&Element;U)$

The probability that described historical data enters ranking list obtains by following manner:

$P\left(A\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}\left(t_{\mathrm{ij}}\right)}{n},(t_{\mathrm{ij}}\&Element;U)$

Described historical data obtains by following manner less than the probability of seniority among brothers and sisters day data:

$P\left(B\right)=\frac{\underset{j=\mathrm{1,2},...,n}{\mathrm{count}}({t_{\mathrm{ij}}}^{\&prime;}<x_{i1})}{n},(t_{\mathrm{ij}}\&Element;U,{t_{\mathrm{ij}}}^{\&prime;}\&Element;(U\&cup;V))$

The list probability that advances of described seniority among brothers and sisters day data obtains by following manner: $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)};$

Wherein, x _I1Be the seniority among brothers and sisters day data of i kind objects of statistics, t _IjBe i kind objects of statistics in the data of j the day of trade, U be historical data enter ranking list advance the list data set, V be historical data do not advance the list data set, n is the default time period.

11. equipment as claimed in claim 8 is characterized in that, described computing unit specifically is used for,

Probability and the data difference general probability of the probability of valid data difference, the data difference when historical data enters ranking list when it is calculated that with the seniority among brothers and sisters number of days obtaining historical data enters ranking list according to described historical data; The probability of the probability of valid data difference, the data difference when historical data enters ranking list and data difference general probability when entering ranking list according to described historical data, that determines seniority among brothers and sisters day data advances the list probability.

12. equipment as claimed in claim 11 is characterized in that, the probability of valid data difference obtained by following manner when described historical data entered ranking list:

$P\left(B\right|A)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}},(t_{\mathrm{ij}}\&Element;U);$

The probability of the data difference when described historical data enters ranking list obtains by following manner:

$P\left(A\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

Described data difference general probability obtains by following manner:

$P\left(B\right)=\frac{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}\&CenterDot;\frac{r_{\mathrm{ij}}}{r_i}}{\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-t_{\mathrm{ij}})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{(t_{\mathrm{ij}}-x_{i1})}+\underset{j=1,x_{i1}>t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}(x_{i1}-{t_{\mathrm{ij}}}^{\&prime;})+\underset{j=1,x_{i1}<t_{\mathrm{ij}}}{\overset{n}{\mathrm{\&Sigma;}}}\frac{1}{({t_{\mathrm{ij}}}^{\&prime;}-x_{i1})}},$

(t _ij∈U，t _ij′∈(U∪V))；

The list probability that advances of described seniority among brothers and sisters day data obtains by following manner: $P\left(A\right|B)=\frac{P(A,B)}{P\left(B\right)}=\frac{P\left(B\right|A)\&CenterDot;P\left(A\right)}{P\left(B\right)};$

Wherein, x _I1Be the data of described i kind objects of statistics on same day seniority among brothers and sisters day, t _IjBe i objects of statistics in the data of j the day of trade, r _IjBe t _IjRank in the data sample that the data of part objects of statistics form in the j row, r _iBe x _I1Rank in the data sample that the data of seniority among brothers and sisters part objects of statistics on day same day form, U be historical data enter ranking list advance the list data set, V be historical data do not advance the list data set, n is the time period.

13. equipment as claimed in claim 8 is characterized in that, described processing unit specifically is used for,

According to the non-damaged data in the raw data in the time period, calculate the entropy of each noncolobomatous day of trade of data; According to the entropy of the described day of trade, calculate the coefficient of variation in the described day of trade; According to the coefficient of variation of the described day of trade, calculate the weight coefficient of the described day of trade;

Weight coefficient according to each the noncolobomatous day of trade of data that calculates calculates damaged value.

14. equipment as claimed in claim 13 is characterized in that, the damaged value of described calculating comprises:

Calculate the entropy I of f the day of trade _f=-kp _fLn (p _f);

Wherein, k=1/ln (n),

N participates in the time period of the time span of ordering for expression;

Wherein,

d _fBe used for f the day of trade of expression data and the distance between the damaged data; t _IfFor the value f the day of trade i kind objects of statistics data, t _IbFor b the day of trade i kind objects of statistics data;

The described coefficient of variation that calculates in the described day of trade according to the entropy of the described day of trade comprises:

r _f=1-I _f,, f=1 wherein, 2 ..., n;

The weight coefficient that described coefficient of variation according to the described day of trade calculates the described day of trade comprises:

$w_f=\frac{r_f}{\underset{i=1}{\overset{m}{\mathrm{\&Sigma;}}}},f=\mathrm{1,2},...,m$

The weight coefficient of each noncolobomatous day of trade of data that described basis calculates calculates damaged value and comprises:

t _ab＝w ₁t _a1+w ₂t _a2+...w _(b-1)t _a(b-1)+w _(b+1)t _a(b+1)+...+w _nt _an

Wherein, t _AbBe damaged value.

Images