TRANSFER LEARNING FOR RESOLVING SPARSITY PROBLEM IN RECOMMENDER SYSTEMS: HUMAN VALUES APPROACH

  • Abhishek Srivastava Indian Institute of Management
  • Pradip Kumar Bala Indian Institute of Management
  • Bipul Kumar Indian Institute of Management
Keywords: Recommender systems, Collaborative filtering, Sparsity problem, Transfer learning, Basic Human Values

Abstract

With the rapid rise in popularity of ecommerce application, Recommender Systems are being widely used by them to predict the response that a user will give to a given item. This prediction helps in cross selling, upselling and to increase the loyalty of their customers. However due to lack of sufficient feedback data these systems suffer from sparsity problem which leads to decline in their prediction efficiency. In this work, we have proposed and empirically demonstrated how the Transfer Learning approach using five dimensions of basic human values can be successfully used to alleviate the sparsity problem and increase the efficiency of recommender system algorithms.

Downloads

Download data is not yet available.
Published
2017-12-01
How to Cite
Srivastava, A., Bala, P., & Kumar, B. (2017). TRANSFER LEARNING FOR RESOLVING SPARSITY PROBLEM IN RECOMMENDER SYSTEMS: HUMAN VALUES APPROACH. JISTEM - Journal of Information Systems and Technology Management (Online), 14(3), 323-337. https://doi.org/10.4301/s1807-17752017000300002
Section
Articles

Introduction

With the rapid rise in popularity of e-commerce applications and the dramatic increase in the size of data present in such applications along with the related social media data generated by customers, information filtering technique like Recommender Systems( RS) are being widely used to help the customer in finding the item they might find useful. This prediction also helps the companies in cross selling, upselling and to increase the loyalty of their customers. Recommender Systems [RS] have emerged as an important area of research and its rise can be traced to its practical importance in the area of ecommerce. 60% of the netflix rentals, 35% of the sales of Amazon, 38% increase in article view of Google news can be attributed to the recommendations made by RS algorithms (Hosanagar & Fleder, 2013) (Das, Datar, Garg, & Rajaram, 2007). RS primarily envisages the level of inclination of a given item by a user, based on their prior interaction with the system or on the basis of the behavior of other users and makes the relevant recommendations. This level of inclination is often expressed in the form of explicit item ratings, their reviews and comments expressed in social media channels like blogs , forums etc. , by their usage pattern of the item and purchase behavior . Various approaches to recommender system make use of user’s prior ratings, their demographic profiles, purchase history and their Big Five personality profiles etc. Two common mechanisms used by RS are Collaborative Filtering (Resnick, Iacovou, & Suchak, 1994) and Content Based approach(Lops, De Gemmis, & Semeraro, 2011) .

Collaborative Filtering (CF) algorithms, most widely used technique in RS, are primarily based on the phenomenon of homophily which means that humans tend to associate and bond with other humans who are similar to them. This also gets reflected in their preferences for products and the feedback they give to them. A typical Collaborative Filtering framework is designed like a utility matrix which involves a two-dimensional user-item matrix and it consists of ratings provided by users to the items. Similarity between users in CF is primarily measured through the inverse of distance between their rating vectors. Since in real world, each user generally provides rating for only a fraction of the total items present in the system, hence most of the user-item pair in the utility matrix remains unrated. This leads to data sparsity probem and impacts the quality of recommendations . In the commercial RS , density of the available data can be up to just 1 % and hence data sparsity is one the key challenge for CF based approaches (Basu, Hirsh, & Cohen, 1998).

Transfer Learning (TL) is a machine learning paradigm that makes use of knowledge learned in one task in a different but related source domain, to solve the task in other target domain. This is particularly useful when there is scarcity of fewer high-quality training data. In order to resolve this sparsity problem in RS, we propose a TL based approach based on Basic Human Values .

To the best of our knowledge our work presented in this paper contributes to the existing body of knowledge in the academic and practical domain of Recommender system by proposing and empirically demonstrating for the first time the TL approach based on Basic Human Values, as a solution to alleviate the sparisty problem in a collaborative filtering based recommender system . The way we have implemented it by directly extracting the Human Values through the usage of advancement in linguistic analytics also makes it easier to implement the solution for practical purpose in industry.

The remainder of the paper is structured in the following manner. In section 2 we first review the existing literature on RS and various approaches, followed by the discussion of details about Basic Human values, their various dimensions and how it has been shown that they impact consumer’s choices. Section 3 describes the model development and how similarity has been calculated in the proposed model. In section 4 we explain the experimental setup that has been used to empirically verify the usefulness of Human Values in improving the efficiency of Recommender system algorithms. Section 4.1 refers to acquisition of data sets and extraction of basic human values from it, section 4.2 describes how the data sets have been partitioned into train and test sets, at various levels of sparsity and different neighborhood sizes. In sec 4.3 we describe the various evaluation measures used to compare the proposed model with that of traditional collaborative filtering algorithm. In Section 5 we discuss the results of our experiments in detail, followed by section 6 where we summarize our work and provide the direction for future research.

RELATED WORK

The term collaborative filtering was coined in early 90’s (D. Goldberg, Nichols, Oki, & Terry, 1992) (Su & Khoshgoftaar, 2009) and one of the first work using this technique was done in mid-90’s (Resnick et al., 1994) (Park, Kim, Choi, & Kim, 2012) using the group lens movie data . Collaborative filtering algorithm works on the basic assumption that if two individuals A and B like similar items or give similar rating to n items, then they will do similarly for other items too (K. Goldberg, Roeder, Gupta, & Perkins, 2001). A typical Collaborative filtering frame work involves a two dimensional use-item or utility matrix that consists of ratings given by a set of users for set of items. Distance between users is calculated between various users and accordingly the users are arranged in neighbored depending on their distances with each other. Ratings given by the closest neighbor by an individual are then used to predict the ratings for items which that particular individual has not rated yet. As the user-item utility matrix is usually sparse , it leads to data sparsity problem.

Content based approach creates a user profile for each user by analyzing the content of items that have been rated by the user and then uses this profile to make future recommendations (Basu et al., 1998). A user profile contains the description of items that have been evaluated by the user and history of his interaction with the system.

Both the filtering techniques i.e. collaborative and content based filtering are often merged together to utilize advantages of both and this has given rise to hybrid system (Burke, 2002) . Key issues associated with the existing Recommender system algorithms include - Sparsity problem, Cold Start problem, Over specialized recommendations etc (Claypool, Gokhale, & Miranda, 1999) (Blanco-Fernández & Pazos-Arias, 2008)

Transfer learning is motivated by the fact that people can intelligently apply knowledge learned previously to solve new problems faster or with better solutions (Pan & Yang, 2010) . In the context of RS , TL based approach using psychological aspects of the individuals that get reflected in their personality attributes have been used (Hu & Pu, 2011) (Tkalčič, Kunaver, Tasič, & Košir, 2009) . Five Factor Model or Big Five has emerged as the instrument of choice to measure personality by researchers when compared to the older MBTI model (McCrae & Costa, 1989) (McCrae & Jr, 2013) . The five factors are - Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism (McCare & John, 1992) . Apart from this demographic profiles based on age , gender etc. too has been used , but so far no work has been done in empirically demonstrating the benefits of usage of Human values to address the data sparsity problem in RS(Badenes et al., 2014).

Values of a person express what is important to him/her in life (e.g., security, independence, wisdom, success, kindness, pleasure) . In social sciences human values have been considered to be an integral factor in elucidating the social and personal organization and changes by Durkheim (Durkheim, 1893) (Durkheim, 1897) (Durkheim, 1912) and (Weber, 1958) . They have been shown to impact consumer behavior (Puohiniemi, 1995) , environmental attitudes and intention to purchase organic foods (Grunert & Juhl, 1995) , trust and conflict (Devos, Spini, & Schwartz, 2002) , religiosity (Schwartz & Huismans, 1995). Various models to measure Human values have been suggested , (Kluckhohn, 1951) (Kohn, 1969) (Rokeach, 1973) (Schwartz, 1994) (Inglehart, 1997) (Boudon, 2001) (Chen, Hsieh, Mahmud, & Nichols, 2014) among which Schwartz‘s theory of basic human values have recognized throughout all the major culture (Berry, Poortinga, & Pandey, 1997) .

As defined by Schwartz values are “desirable, transsituational goals, varying in importance, that serve as guiding principles in people's lives”( (Schwartz, 1996).Its six key features are -

  1. Values are beliefs linked inextricably to affect

  2. Values refer to desirable goals that motivate action.

  3. Values transcend specific actions and situations.

  4. Values serve as standards or criteria.

  5. Values are ordered by importance relative to one another.

  6. The relative importance of multiple values guides action.

There are 10 Basic Human Values which can be mapped to 5 higher dimensions as depicted in fig 1 , the defining goal of each in summarized in Table 1.

Five dimensions of basic human values

Figure 1: Five dimensions of basic human values

Table 1:
Basic Human Values and their defining goal
Value Defining Goal
Self-Direction independent thought and action--choosing, creating, exploring
Stimulation excitement, novelty, and challenge in life
Hedonism pleasure or sensuous gratification for oneself
Achievement personal success through demonstrating competence according to social standards
Power Social status and prestige, control or dominance over people and resources.
Security Safety, harmony, and stability of society, of relationships, and of self.
Conformity restraint of actions, inclinations, and impulses likely to upset or harm others and violate social expectations or norms
Tradition respect, commitment, and acceptance of the customs and ideas that one's culture or religion provides
Benevolence Preserving and enhancing the welfare of those with whom one is in frequent personal contact
Universalism Understanding, appreciation, tolerance, and protection for the welfare of all people and for nature.

Model Development

This section gives detailed steps of the model proposed in this paper. First we discuss the existing model used in traditional collaborative filtering which is used as a base to develop our model which is discussed in the next section

Neighborhood model (k-NN model)

The most common approach in collaborative filtering is based on neighborhood model or k-nearest neighbor model (k-NN). The basic concept of this model derives from the assumption that like-minded users tend to rate movies in similar manner. So for any item ‘i’ if one needs to predict the rating for a user ‘u’ the likeminded users need to be identified based on the items which are rated by all the other users. The prediction of rating for an unseen item ‘i’ for an active user ‘u’ is then computed. The degree of like mindedness between users is often termed as similarity measures which is mostly cosine similarity or Pearson correlation based similarity. The similarity measure acts as weights () between like-minded users (user u and user v) which are normalized by dividing the sum of product of weight and rating by the total sum of weights. A set of predicted rating is given by:

  r ^ ui = r ¯ u + j S κ ( v;i )       w u v   * ( r vi r ¯ v ) j S κ ( v;i )        w uv  

Here is set of all users who have rated item i, is rating of a user v who has rated item i. and are average rating provided by user u and user v respectively.() can be calculated either by cosine similarity measure or Pearson correlation coefficient.

   w uv   (cosine similarity)=   i=1 n r ui  . r vi i=1 n r ui 2 i=1 n r vi 2

    w uv   (Pearsoncorrelation coeffiecient)  =   i=1 n ( r ui r ¯ u )( r vi r ¯ v ) i=1 n ( r ui r ¯ u ) 2 i=1 n ( r vi v) 2

There are other types of similarity measures which can be used. Like, user-user similarity, item-item similarity can also be applied to predict the rating of an unrated item i by a user u. Similarity measures such as cosine measure and Pearson correlation coefficient can be used to obtain k nearest items similar to the item which has to be rated by user u. In our model we have used this widley common cosine similarity.

Proposed value based neighborhood model

The classical neighborhood model as depicted in Fig. 2 deteriorates as the sparseness in dataset increases. The problem is termed as sparsity problem. To alleviate the sparsity problem we propose a TL based approach, that makes use of Basic Human Values based neighborhood model as depicted in Fig 3 .

Traditional Collaborative Filtering Model

Figure 2: Traditional Collaborative Filtering Model

TL using Basic Human Values based proposed model

Figure 3: TL using Basic Human Values based proposed model

The Values can be derived from social media data of the users, and in our case we have used the text reviews given by the users. This additional knowledge learned from the auxiliary data is then transferred to perform the primary task of recommendation. This ensures that even if data is sparse, the model is not dependent on just those data points to calculate the neighbors. In classical k-NN approach the ratings of user for items is used for calculating similarity between users, while in proposed approach the values determined using reviews is used to calculate similarity between to users. A prediction function given in equation (1) is used to predict the ratings of unseen products. For calculating the similarities we can use either of cosine or Pearson correlation coefficient.

w uv   (cosine similarity) = i=1 n p ui  . p vi i=1 n p ui 2 i=1 n p vi 2

  w uv   (Pearsoncorrelation coeffiecient) =  i=1 n ( p ui p u ¯ )( p vi p v ¯ ) i=1 n ( p ui p u ¯ ) 2 i=1 n ( p vi p v ¯ ) 2

Here, and are elements of value determined using text review for user u and v respectively, n is number of elements that depicts the value of each user.

EXPERIMENTAL STUDY

Data acquisition

In order to empirically study and verify our proposed model we have used the Amazon movie reviews data set which has been shared at Stanford Large Network Dataset Collection as part of Stanford Network Analysis Platform (SNAP) of Stanford university . This is a publicly available data and has been earlier used other research work in RS domain (McAuley & Targett, 2015)(McAuley, Pandey, & Leskovec, 2015). For our experimental purpose we have extracted the reviews for top 1000 rated movies by random 95 users. The ratings are in the scale of 1 to 5 after the extraction of reviews, the text was cleaned to html formatting tags and junk characters. Then we have used user modeling services of IBM Watson personality insight service by creating a bluemix instance to extract the Basic Human Value dimensions of each user on the basis of their reviews. It utilizes primarily the work in linguistic analytics done by (Chen et al., 2014) .The algorithm has used the already established Portrait Value Questionnaire (Schwartz, 2003) to validate its result . The output gives us five value dimensions in percentiles - Conservation, Openness to change, Hedonism, Self Enhancement, Self-transcendence.

Experimental Runs

Cross-validation (Kumar, Bala, & Srivastava, 2016) is used to assess the performance of proposed model over the classical neighborhood approach on various performance measures. The data set is partitioned into 5 equal disjoint sets with 4 datasets used for training and one left out dataset for testing the model. The process is repeated five times, as a procedure adopted for cross validation, and each time the training and test dataset used would be disjoint with 4 dataset used for training and left out to train. Since proposed model is expected to give a better result over classical neighbourhood model for cold start problem, sparsity has been induced in the dataset. The level of sparsity in the original dataset is 80% which has been increased for experimentation at sparsity level of 85, 90, 95, 96, 97, 98 and 99%. In order to compare both the model on the basis of various evaluation measures, neighborhood size (k) has been varied from 5 to 35 in steps of 5 for both the models and the evaluation measures have been recorded at various sparsity level.

Evaluation metrics

In order to evaluate accuracy, the RMSE (Root Mean Square Error) and MAE (Mean Absolute Error) are popular metrics in RS domain research (Kumar, 2016). Since, RMSE gives more weightage to larger values of errors while MAE gives equal weightage to all values of errors, RMSE is preferred over MAE while evaluating the performance of RS. RMSE is popular metrics in RS until very recently and many previous works have based their findings on this metrics, therefore this metrics has been used primarily to exhibit the performance of the proposed models and RSVD model on various datasets.

For a test user item matrix ‘(’ the predicted rating for user-item pairs (u, i) for which true item rating are known, the RMSE is given by

RMSE = 1   | |   ( u,i ) (  r ^ ui   r ui   ) 2

MAE on the other hand is given by

MAE= 1   | |   ( u,i ) |  r ^ ui   r ui |

However, when the task is to find good relevant items for recommendations, the MAE and RMSE metric might not be sufficient to evaluate the overall performance of an RS . Therefore, different accuracy metrics like Precision , Recall and F1 measure are used which consider the frequency with which a RS makes correct or incorrect decisions (classifications) about whether an item is good or not (Tyagi & Bharadwaj, 2013). Precision is the ratio of correctly predicted items to the total number of items recommended to the user as given by equation (8).

Precision = number of correctly recomended items total recommended items

Recall is the ratio of items predicted correctly to the total number of items that are actually preferred by the user as shown in equation (9).

Recall = number of correctly recommended items total items liked by user

F1-measure is the weighted average of precision and recall as given by equation (10).

F1-measure = 2nprecisionnrecall precisionþrecal

5. Results and Discussion

In this section we present the results of the various experimental runs performed in section 5 and discussion its interpretations. MAE and RMSE values of the traditional and proposed models at various sparisty levels have been shown in Table 2 and graphically depicted in Figure 4 and 5 . As both MAE and RMSE are error measures, we expect our proposed model to have a lower value than the traditional model , for it to be considered a better one. As seen in Table 2 and Figure 4 and 5 , both MAE and RMSE values are lower for the proposed model than the traditional model, and the difference continues to increase as the sparsity of data is increased, particularly beyond 95% . On an average, the proposed model shows a 7.84% improvement in MAE and 6.12% improvement in RMSE over the traditional one across the various sparsity levels from 80% to 99%.

Higher values of evaluation metrics of Precision, Recall and F1 measures indicate better performance of a model. As shown in Table 3 - 5 and Figure 6-11, both the models detoriate in performance at sparsity levels is increased , but our proposed model still outperforms traditional models for both top 5 and 10 recommendations consistently.

Table 2:
MAE and RMSE at different Sparsity Level
Sparsity % MAE_Traditional Model MAE Proposed Model RMSE Traditional Model RMSE Proposed Model
80 0.628545209 0.636562104 0.840795043 0.848672753
85 0.655992636 0.660842003 0.87568916 0.878902697
90 0.677086864 0.67306645 0.905027626 0.897071612
95 0.735483466 0.712853471 0.990098643 0.959124849
96 0.783864745 0.7366094 1.046836263 0.98989159
97 0.860027455 0.7488405 1.129117705 1.009658098
98 1.03670117 0.866157436 1.305145056 1.136599866
99 1.288322542 0.957406431 1.485669938 1.228992012

Table 3:
Precision for 5 and 10 recommendations at different Sparsity Level
Sparsity % Precision_5 Traditional Model Precision_5 Proposed Model Precision_10 Traditional Model Precision_10 Proposed Model
80 0.664422388 0.685850817 0.691934323 0.689588426
85 0.641703275 0.651968997 0.678463778 0.678131995
90 0.665792008 0.672231255 0.660809255 0.661584248
95 0.649829764 0.654961295 0.580248654 0.583176702
96 0.589832777 0.621553675 0.537240269 0.564035327
97 0.47318599 0.575101616 0.436581701 0.534757716
98 0.256926952 0.480240922 0.251655629 0.458581505
99 0.070166928 0.380415677 0.070533074 0.376151052

Table 4:
Recall for 5 and 10 recommendations at different Sparsity Level
Sparsity % Recall_5 Traditional Model Recal_5 Proposed Model Recall_10 Traditional Model Recal_10 Proposed Model
80 0.015271725 0.017072546 0.062644194 0.062874904
85 0.028859566 0.030933082 0.114712535 0.116107446
90 0.057823992 0.057988841 0.162363683 0.158737002
95 0.129977215 0.132993659 0.229420491 0.228940217
96 0.151499167 0.159022765 0.229844531 0.241282621
97 0.157856626 0.190731354 0.198696597 0.243410572
98 0.105263158 0.189928058 0.117647059 0.21294964
99 0.021122112 0.16039604 0.031023102 0.165346535

Table 5:
F1 Measure for 5 and 10 recommendations at different Sparsity Level
Sparsity % F1Measure_5 Traditional Model F1Measure_5 Proposed Model F1Measure_10 Traditional Model F1Measure_10 Proposed Model
80 0.029857184 0.033315778 0.114887097 0.11524231
85 0.055235027 0.059063843 0.19624464 0.198268104
90 0.10640658 0.106767567 0.260677725 0.256040911
95 0.216625563 0.221093094 0.328827971 0.328801427
96 0.241077361 0.253251774 0.321950683 0.337983084
97 0.236736923 0.286459095 0.273100147 0.334543783
98 0.149341142 0.272203663 0.160337553 0.290842107
99 0.032469916 0.225650318 0.0430925 0.22971579

MAE at different Sparsity Level

Figure 4: MAE at different Sparsity Level

RMSE at different Sparsity Level

Figure 5: RMSE at different Sparsity Level

Precision for 5 recommendations at different Sparsity Level

Figure 6: Precision for 5 recommendations at different Sparsity Level

Precision for 10 recommendations at different Sparsity Level

Figure 7: Precision for 10 recommendations at different Sparsity Level

Recall for 5 recommendations at different Sparsity Level

Figure 8: Recall for 5 recommendations at different Sparsity Level

Recall for 10 recommendations at different Sparsity Level

Figure 9 : Recall for 10 recommendations at different Sparsity Level

F1 Measure for 5 recommendations at different Sparsity Level

Figure 10: F1 Measure for 5 recommendations at different Sparsity Level

F1 Measure for 10 recommendations at different Sparsity Level

Figure 11: F1 Measure for 10 recommendations at different Sparsity Level

CONCLUSION AND FUTURE DIRECTION

On the basis of the experiments conducted and the results obtained, we have successfully demonstrated empirically that TL approach using Basic Human Values can not only be used in to alleviate sparsity problem in RS , but they also serve as more useful criteria for finding similarities between various users when compared to using the ratings as in traditional collaborative filtering algorithm. Problem of data sparsity which is a major concern in collaborative filtering algorithms considerably less in this this approach when compared to traditional collaborative filtering algorithms. Further the way we have approached this implementation, i.e. direct usage of linguistic analytics to extract the Basic Human Values of users makes it more practical to implement in real world applications when compared to using the surveys to first extract the desired information about the users, which is usually unwieldy in ecommerce applications.

For further research , the basic human values that have been used in this model can be combined with other auxiliary about user and also incorporate the existing ratings as in traditional Collaborative filtering algorithms to further improve the efficiency of the collaborative filtering based Recommender system algorithms . Future research can also be done in content based Recommender system algorithms to create user profiles on the basis of the TL from values.

References

  1. , , , , , , (). System U : Automatically Deriving Personality Traits from Social Media for People Recommendation. Proceedings of the 8th ACM Conference on Recommender Systems 373-374. https://doi.org/10.1145/2645710.2645719
  2. , , (). . Recommendation as classification: Using social and content-based information in recommendation. AAAI/IAAI.
  3. , , (). . . .
  4. , (). A flexible semantic inference methodology to reason about user preferences in knowledge-based recommender systems. Knowledge-Based …
  5. (). . . Transaction Publishers. .
  6. (). . . . https://doi.org/10.1023/A:1021240730564
  7. , , , .. 405-414. https://doi.org/10.1145/2531602.2531608
  8. , , ..
  9. , , , .. 271-280. https://doi.org/10.1145/1242572.1242610
  10. , , (). Conflicts among human values and trust in institutions. Journal of Social Psychology https://doi.org/10.1348/014466602321149849
  11. (). . . New York: Free Press. . G. Simpson, Trans
  12. (). . . Glencoe, IL: Free Press. .
  13. (). . . New York: Free Press. . 1912Trans. Karen E. Fields
  14. , , , (). Using collaborative filtering to weave an information tapestry. Communications of the ACM 35(12), 61-70. https://doi.org/10.1145/138859.138867
  15. , , , (). Eigentaste: A constant time collaborative filtering algorithm. Information Retrieval 4(2), 133-151. https://doi.org/10.1023/A:1011419012209
  16. , (). Values, environmental attitudes, and buying of organic foods. Journal of Economic Psychology 16(1), 39-62.
  17. , (). Will the global village fracture into tribes? Recommender systems and their effects on consumer fragmentation. Management Science
  18. , .. 1997. https://doi.org/10.1145/2043932.2043969
  19. (). . . .
  20. (). . . .
  21. (). . . . IL. IU., Dorsey. 4
  22. (). A novel latent factor model for recommender system. JISTEM-Journal of Information Systems and Technology Management 13(3), 497-514. https://doi.org/10.4301/s1807-17752016000300008
  23. , , (). Cosine Based Latent Factor Model for Precision Oriented Recommendation. INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS 7(1), 451-457.
  24. , , (). Content-based recommender systems: State of the art and trends. Recommender systems handbook 73-105.
  25. , , (). Inferring networks of substitutable and complementary products. Proceedings of the 21th ACM …
  26. , (). Image-based recommendations on styles and substitutes. Proceedings of the 38th …
  27. , (). An introduction to the five‐factor model and its applications. Journal of personality 60(2), 175-215.
  28. , (). Reinterpreting the Myers‐Briggs type indicator from the perspective of the five‐factor model of personality. Journal of Personality
  29. , (). . Introduction to the empirical and theoretical status of the five-factor model of personality traits.
  30. , (). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering 22(10), 1345-1359.
  31. , , , (). A literature review and classification of recommender systems research. Expert Systems with Applications https://doi.org/10.1016/j.eswa.2012.02.038
  32. , (Compiler), (Compiler), (Compiler) ..
  33. (). . . .
  34. (). Are There Universal Aspects in the Structure and. Journal of Social Issues
  35. (). A proposal for measuring value orientations across nations. Package of the European Social Survey
  36. , (). Value priorities and religiosity in four western religions. Social Psychology Quterly
  37. , (). A Survey of Collaborative Filtering Techniques. Advances in Artificial Intelligence, 2009 (Section 3) 1-19. https://doi.org/10.1155/2009/421425
  38. , , , (). Personality Based User Similarity Measure for a Collaborative Recommender System. 5th Workshop on Emotion in HumanComputer InteractionReal World Challenges, 30
  39. , (). Enhancing collaborative filtering recommendations by utilizing multi-objective particle swarm optimization embedded association rule mining. Swarm and Evolutionary Computation 13, 1-12. https://doi.org/10.1016/j.swevo.2013.07.001
  40. (). . . New York: Charles Scribner’s Sons. . trans. Talcott Parsons