Class Summary

☀️ Quick Takes

Is this Class Clickbait?

1-Sentence-Summary

Favorite Quote from the Author

your goal is to hear minority voices your goal is to ensure that smaller groups are not overshadowed by the larger ones

💨 tl;dr

Sampling is crucial for research, involving selecting a smaller group from a larger population. Different strategies exist, like convenience and purposive sampling, each with pros and cons. A solid sampling frame is key to avoid bias and ensure valid results.

💡 Key Ideas

• Sampling is the process of selecting a smaller group (sample) from a larger group (population) for research purposes.
• A sampling frame represents the population but is often incomplete or unreliable.
• Different sampling strategies include non-probabilistic (e.g., convenience, purposive, snowball) and probabilistic (e.g., simple random, systematic, stratified).
• Convenience sampling is quick and low-cost but may produce biased results and lacks generalizability.
• Purposive sampling uses researcher judgment to select items based on specific characteristics relevant to the study.
• Snowball sampling builds on referrals from initial participants, useful for accessing hidden populations but can introduce bias.
• Respondent-driven sampling prevents duplicate selections and involves diverse initial participants to reduce bias.
• Random sampling gives each item an equal chance of inclusion, but external factors can compromise its integrity.
• Stratified sampling enhances diversity by dividing samples into groups and ensuring representation from each.
• Cluster sampling involves selecting from groups within the sampling frame, which can be adjusted based on findings.
• Panel sampling is a repeated study of the same subjects, often used in educational research.
• Sampling strategies should align with research objectives and the nature of the population being studied.

🎓 Lessons Learnt

• Sampling is essential in all research strategies. It's a fundamental aspect that affects the validity and reliability of research findings.

• Be cautious with convenience sampling. While it's fast and cheap, it can lead to biased results if the sample doesn't accurately represent the population.

• Understand the importance of a proper sampling frame. Without a clear and complete list of the population, sampling becomes challenging and can skew results.

• Employ diverse sampling methods for richer insights. Combining different sampling strategies (like purposive and random sampling) can yield more meaningful data.

• Utilize purposive sampling for specific goals. This method allows researchers to target participants based on relevant characteristics, enhancing data quality.

• Snowball sampling can enhance literature reviews. It helps uncover hidden sources by exploring references in identified papers, but it can be time-consuming.

• Recruitment strategies matter. Use diverse initial participants and limit recruitment numbers to avoid selection bias and ensure balanced representation.

• Stratified sampling improves representativeness. It intentionally selects groups to highlight minority experiences, preventing majority bias.

• Random selection must truly be random. Avoid biases in selecting participants to ensure the sample is representative of the population.

• Be aware of the limitations of sampling methods. Different methods have inherent biases and may not guarantee diverse representation, so context matters.

🌚 Conclusion

In summary, effective sampling enhances research quality. Be mindful of biases, use diverse methods, and ensure your sampling frame is reliable to get accurate insights.

In-Depth

All Key Ideas

Sampling in Research

• The lecture focuses on selecting people, companies, projects, or artifacts for study.
• Sampling is defined as the process of selecting a smaller group of items to study from a larger group, known as the population.
• All research strategies require sampling, whether in field studies, laboratory experiments, or theoretical derivations.
• Key distinctions in sampling include population, sampling frame, and sample.

Sampling in Software Engineering

• A population is a larger group of interest represented by an imperfect list known as a sampling frame.
• A sample consists of individuals selected from the sampling frame.
• Sampling in software engineering can be challenging due to the lack of a complete list of open source projects.
• Identifying samples on platforms like GitHub is complicated due to multiple aliases and overlapping repositories.
• There are different sampling strategies, categorized into non-probabilistic, probabilistic, and multi-stage sampling.
• Convenience sampling involves selecting items based on their availability or ease of study.
• Convenience sampling is common in software engineering research due to its speed and low cost.

Sampling Methods in Research

• Sampling frame is a representation of the entire population, but obtaining a complete list is often impossible.
• Convenience sampling alleviates the need for a sampling frame but lacks statistical generalizability and may produce biased results.
• Convenience sampling is popular in laboratory experiments and exploratory studies but controversial.
• The purpose of purposive sampling is assembling items based on specific characteristics relevant to the study's objective.
• Non-probability sampling aims to find accessible and information-rich cases relevant to the study.
• Heterogeneity sampling focuses on selecting projects that are diverse as a whole rather than individually.
• Creative-based sampling can involve specific search queries to extract samples from large repositories.

Purposive and Snowball Sampling

• This is what is known as expert sampling; it is a form of purposive sampling.
• The advantages of purposive sampling include the researcher exercising expert judgment and ensuring representativeness of the sample regarding specific domains or dimensions.
• Limitations of purposive sampling include lack of statistical generalizability and intrinsic subjectivity.
• Purposive sampling is beneficial for exploratory studies and studies of universal phenomena.
• Snowball sampling involves selecting items based on their relations to previously selected items, starting with a small group and growing from there.

Snowball Sampling Overview

• Snowball sampling is commonly used in literature studies to supplement query-based sampling by exploring references cited in papers.
• Backward snowballing involves checking the papers cited by a focus paper to identify relevant ones, while forward snowballing checks which papers cite the focus paper.
• The process of snowball sampling can be labor-intensive and continues until no more relevant papers are found.
• Snowball sampling allows researchers to go beyond a perfect sampling frame and reach hidden populations without available lists.
• A bias in snowball sampling exists towards better-connected items in a population, potentially skewing the results based on referral links.

Respondent-Driven Sampling

• Respondent-driven sampling avoids selecting individuals multiple times by ensuring each person can only be selected once.
• A diverse set of initial participants is crucial for effective sampling; they should not know each other and should represent different sub-populations.
• Limiting the number of referrals each participant can make increases the number of recruitment waves and reduces bias.
• The process continues until the sample reaches an equilibrium where the distribution of variables stabilizes.
• Respondent-driven sampling, while similar to snowball sampling, addresses its shortcomings and allows access to hidden populations.
• Both respondent-driven and snowball sampling violate the premise of statistical analysis, making traditional techniques inapplicable.

Sampling and Application Discussion

• The application selected for discussion has a purpose that extends beyond availability, indicated by its extensive documentation and the number of VMs.
• The sampling process includes a two-stage approach: purposive sampling for the identification of presenters, followed by snowball sampling through referrals.
• Probabilistic sampling requires a sampling frame and true random selection; without these, it cannot be considered random.
• Random sampling can be compromised by various external factors, such as time of day or platform limitations in recruiting participants.
• Whole frame sampling involves taking the entire population as a sample, which can be debatable as probabilistic sampling, especially in small sampling frames.

Sampling Methods and Considerations

• Whole frame sampling is advantageous when the sampling frame is reliable and represents the population, but it's not feasible for large populations due to time and effort constraints.
• Inadequate sampling frames lead to unreliable results; for example, using public GitHub repositories to derive conclusions about all GitHub repositories is inadequate.
• Simple random sampling involves selecting items entirely by chance, giving each item an equal chance of inclusion.
• A random sample can be determined by allocating numbers to individuals in a population and using a random number generator to select participants.
• Statistical methods exist to determine sampling size based on population size, confidence level, and confidence interval.
• Confidence interval represents the margin of error in survey results, indicating the range in which the true percentage likely falls.

Advantages and Considerations of Sampling Methods

• The major advantage of simple random sampling is statistical generalizability, assuming it is done correctly.
• Representativeness of a sample is the degree to which it resembles the target population, but it can vary across different dimensions.
• Probabilistic sampling does not guarantee representativeness, as randomness can lead to biased samples (e.g., only small or large projects).
• Non-probabilistic sampling can sometimes be more representative concerning specific properties of interest.
• Systematic random sampling involves selecting a random starting point and then choosing every nth item, but bias can occur if the selection rhythm coincides with project patterns.
• Consistent intervals in sampling can lead to different conclusions based on the timing of data collection (e.g., measuring temperature in different months).

Sampling Strategies in Research

• Carl Chapman and Katie Stolle used sampling every 20 companies to avoid describing all commits in every project, ensuring equal probability for inclusion.
• Risks in coin-consistent sampling intervals may coincide with project writing cycles, impacting results.
• A study by Mario Linares Vasquez and co-authors analyzed Android APIs, leading to various interpretations of their sampling strategy—simple random sampling is suggested due to lack of detailed selection information.
• Scientific papers often lack comprehensive detail on sampling strategies due to space limitations or missing information.
• Multi-stage sampling combines two or more sampling strategies, like stratified quota sampling, to ensure proportional representation across different groups.
• In stratified sampling, developers can be divided into strata (e.g., by ethnicity), ensuring diverse representation in the sample.
• The example of 42 participants in a study shows how gender can determine strata in sampling, highlighting the importance of diversity in research sampling.

Sampling Methods in Research

• There were 10 participants sampled among women and 10 among men, with no non-binary participants, indicating quota sampling.
• In quota sampling, participants are selected purposefully, often leading to underrepresentation of majority groups like men.
• Stratified random sampling involves dividing samples based on entity types and randomly selecting from each group.
• Stratified sampling aims to amplify minority voices rather than creating a representative sample.
• Cluster sampling divides the sampling frame into groups and selects items from a subset of those groups.
• Cluster sampling can be random or adaptive, where the sample can be adjusted based on findings.
• The effectiveness of cluster sampling relies on the similarity of strata; heterogeneous strata can complicate representation.
• Adaptive sampling allows for sampling more based on interesting results, similar to snowball sampling.

Sampling Strategies in Research

• The same sample can be studied multiple times regardless of the sampling strategy used (probabilistic, non-probabilistic, or mixed).
• Panel sampling is commonly used in educational contexts to assess knowledge before and after a course.
• Various sampling strategies exist, including non-probabilistic (convenience, purposive, snowball) and probabilistic (simple random, systematic random, stratified random, cluster).
• Research articles often utilize multiple sampling strategies, with a focus on non-probabilistic methods in many cases.
• Field studies require a researcher to be embedded in a project for a profound understanding, necessitating careful project selection.

Sampling Methods in Research

• Communication and interdependency in large projects differ from monolingual projects, suggesting convenience and purposive sampling are suitable.
• Judgment studies require purposive sampling to obtain expert opinions.
• Iso stratified random sampling or quarter sampling is used in lab experiments to ensure diverse participant types.
• Sample studies often combine purposive and random sampling methods based on specific criteria.
• Snowball sampling and respondent-driven sampling are used for studies involving human beings and rely on referrals.
• Sampling is the process of selecting a smaller group (sample) from a larger group (population) of interest.
• Sampling frames may be unreliable or incomplete, impacting research strategies.
• Common non-probabilistic sampling methods include convenience sampling and purposive sampling.
• Respondent-driven sampling improves upon snowball sampling by addressing oversampling issues.
• Probabilistic sampling involves randomness, with simple random and systematic random sampling being common methods.
• Probabilistic and non-probabilistic sampling methods can be combined, such as in stratified sampling or cluster sampling.

Sampling Strategies and Data Collection

• sampling essentially uh depending on um what kind of elements we combine and how those series of decisions influence each other
• panel sampling is uh simply a repeated uh sampling of the same uh items the same people the same projects
• we have seen a couple of mappings of those sampling strategies and research strategies we have discussed last week
• we are going to move to the first actual data collection mechanism and those are interviews
• it’s not only about the questions you ask but also about how do you ask those questions

All Lessons Learnt

Lessons Learnt

• It's normal for some research papers to be difficult to understand.
• Sampling is essential in all research strategies.
• You can reach out for help via Slack effectively.

Sampling Strategies in Research

• Convenience sampling can be practical in research. It's often fast and cheap, allowing researchers to easily recruit participants or select projects based on availability.
• Be aware of the limitations of convenience sampling. While it's easy to implement, it may lead to biased results because the sample may not represent the larger population accurately.
• Identifying a proper sampling frame is crucial. Without a clear and complete list of the population, sampling can become challenging, as seen with open source projects and contributors on platforms like GitHub.
• Understanding different sampling strategies is essential. Depending on the research needs, employing various sampling methods (non-probabilistic, probabilistic, multi-stage) can yield different insights and results.

Sampling Methods in Research

• Convenience sampling has limitations. While it eliminates the need for a sampling frame, it can lead to biased results and lacks statistical generalizability.
• Use convenience sampling for exploratory studies. It’s effective for understanding general phenomena, especially when studying human-related cognitive biases, not just specific technological contexts.
• Purposive sampling should be based on specific characteristics. Selecting items based on criteria relevant to the study’s objective is crucial for gathering meaningful data.
• Heterogeneity sampling can enhance diversity in selection. When aiming for a diverse sample, focus on criteria applicable to the collection as a whole rather than individual projects.
• Creative-based sampling is useful for literature reviews. Using specific queries to extract samples from large repositories can streamline the research process and yield relevant results.

Lessons Learnt about Purposive Sampling

• Purposive sampling allows researchers to leverage expert judgment.
• Representativeness can be ensured in purposive sampling.
• Purposive sampling lacks statistical generalizability.
• Subjectivity is an inherent limitation in purposive sampling.
• Purposive sampling is beneficial for exploratory studies.
• Bias in sampling can yield reliable outcomes depending on context.
• Snowball sampling helps expand a sample based on relationships.

Snowball Sampling in Literature Reviews

• Use snowball sampling to enhance literature reviews. This technique helps to supplement keyword-based searches by exploring references in identified papers, ensuring no relevant sources are overlooked.
• Be prepared for labor-intensive processes. Snowball sampling can be time-consuming as it requires thorough checking of cited and citing papers for relevance, but it's essential for comprehensive literature studies.
• Understand the limitations of sampling frames. Snowball sampling allows researchers to go beyond assumed perfect sampling frames, reaching hidden populations that may not be well-documented or easily accessible.
• Recognize potential biases in snowball sampling. The method may favor better-connected items, leading to an uneven representation of the population, which can skew the understanding of the phenomenon being studied.

Recruitment Strategies

• Use diverse initial participants: Start with a set of seeds that represent different sub-populations and do not know each other to avoid selection bias and enhance peer influence.
• Limit recruitment numbers: Each participant should only recruit a small number of peers (typically 2-3) to prevent bias from highly connected individuals in the sample.
• Require multiple recruitment waves: Aim for about 20 recruitment waves to extend referral chains and reduce the risk of oversampling from a specific subset of the population.
• Continue until equilibrium is reached: Keep the recruitment process going until the distribution of variables stabilizes, indicating that opinions have converged.
• Use advanced statistical techniques: Since respondent-driven sampling does not yield a random sample, apply more sophisticated statistical methods for data analysis instead of classical techniques.

Sampling Methods in Research

• Use purposive sampling for specific goals. When selecting participants for research, ensure there's a clear purpose behind who you choose, as it can lead to more relevant data.
• Be cautious with snowball sampling. Relying on referrals can introduce bias, so be aware of the limitations and lack of control over who gets included in the sample.
• Ensure a proper sampling frame for probabilistic sampling. Without a clear list of the population, random sampling isn't feasible, which is crucial for obtaining accurate results.
• Random selection must truly be random. Avoid selection methods that are influenced by external factors, as they can skew results and won't represent the population effectively.
• Consider the implications of recruitment methods. When advertising for participants, recognize that your chosen platform may exclude certain demographics, affecting the diversity of your sample.

Sampling Techniques

• Use a reliable sampling frame for whole frame analysis. If the sampling frame doesn't accurately represent the population, conclusions drawn will be misleading. For instance, using public GitHub repositories to infer about all GitHub repositories is not adequate.
• Simple random sampling is a feasible alternative. When the entire population can't be surveyed, randomly selecting a sample (like assigning numbers to developers) ensures each member has an equal chance of being included, making the sample representative.
• Determine sample size using confidence level and confidence interval. Knowing the population size and desired confidence metrics helps in calculating how many samples to take to ensure reliable results, crucial for accurate data analysis.

Sampling and Representativeness

• Importance of Adequate Sampling Frames: If the sampling frame is not adequate, any sampling based on it will not produce valid conclusions. The representativeness of a sample depends on the quality of the sampling frame.
• Understanding Representativeness: Representativeness is a complex concept; a sample can be representative in one aspect but not in another. Care must be taken to ensure that all relevant dimensions are considered when assessing representativeness.
• Random Sampling Limitations: Simple random sampling does not guarantee that a sample will include diverse project sizes or types. Bias can occur, leading to non-representative samples.
• Systematic Random Sampling Risks: While systematic random sampling can be useful, its effectiveness relies heavily on the initial random selection. If the starting point is biased, the entire sample can be skewed.
• Context Matters in Sampling: The timing and context of data collection (like measuring weather in different months) can significantly affect outcomes. Consistency in context is crucial for accurate analysis.

Lessons Learnt

• Be cautious with sampling intervals.
• Scientific papers may lack detailed sampling information.
• Use stratified sampling to avoid majority bias.
• Multi-stage sampling combines strategies for better results.

Sampling Methods and Their Importance

• Quota sampling is purposeful: It’s designed to ensure that minority voices are heard, even if it means not being statistically representative.
• Stratified random sampling focuses on differences: This method intentionally selects groups to highlight minority experiences rather than generalize findings.
• Cluster sampling requires careful subgroup selection: Selecting random subgroups is crucial, but they must be similar to avoid bias in the subsequent selection of items.
• Adaptive sampling allows flexibility: This method lets researchers adjust their sampling strategy based on findings, similar to snowball sampling, which can lead to richer data.
• Balancing breadth and depth is important: In sampling, finding a balance between including multiple subgroups and gathering more data from those subgroups is key for effective research.

Lessons Learnt

• You can study the same sample multiple times regardless of the sampling strategy used.
• Not all studies fit standard sampling strategies.
• Combining different sampling strategies can be effective.
• Many articles utilize multiple sampling steps in their research.
• Non-probabilistic sampling is more common than probabilistic sampling in research articles.
• Field studies require careful selection of projects based on access and understanding.

Sampling Methods in Research

• Use convenience and purposive sampling for large projects: In large or multilingual projects, sampling methods like convenience and purposive sampling are effective because they cater to the specific characteristics and needs of the project.
• Expert opinions require purposive sampling: When seeking expert opinions, purposive sampling is essential as it targets individuals with specific expertise rather than random selection.
• Stratified sampling improves lab experiments: For lab experiments, using stratified random sampling or quarter sampling helps ensure diverse participant representation, preventing any single type from dominating the results.
• Combine sampling methods for sample studies: In sample studies, combining purposive elements with random sampling can yield more meaningful data, especially when specific criteria are applied to project selection.
• Respondent-driven sampling addresses oversampling issues: Respondent-driven sampling is a refined approach to snowball sampling that helps mitigate the risk of oversampling well-connected individuals, offering a more balanced sample.
• Understand the importance of a reliable sampling frame: A dependable sampling frame is crucial for accurate sampling; however, it's often unreliable or incomplete, so researchers should be aware of its limitations.
• Mix probabilistic and non-probabilistic sampling methods: Different sampling strategies can be effectively combined, blending probabilistic and non-probabilistic methods to enhance research outcomes.

Lessons Learnt

• Panel sampling is useful for consistency.
• How you ask questions matters.
• Review previous quiz answers for improvement.