Scribd Logo
Upload

Welcome to Scribd!

  • 26 views

    ECE 368 Review Part 1

    Uploaded by

    Jialun Lyu
    This document discusses probabilistic reasoning and includes the following topics: 1. Data processing including data types, training models using supervised and unsupervised learning, and periodic model retraining. 2. Bayesian concepts including Bayes' rule, total probability, and comparing Bayesian and frequentist statistics. 3. Parameter estimation methods including maximum likelihood estimation, maximum a posteriori probability, and Bayesian least mean square estimation. 4. Bayesian inference and examples of parameter estimation for Poisson, Gaussian, and multinomial distributions with conjugate priors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd

    ECE 368 Review Part 1

    Uploaded by

    Jialun Lyu
    26 views55 pages
    This document discusses probabilistic reasoning and includes the following topics: 1. Data processing including data types, training models using supervised and unsupervised learning, and periodic model retraining. 2. Bayesian concepts including Bayes' rule, total probability, and comparing Bayesian and frequentist statistics. 3. Parameter estimation methods including maximum likelihood estimation, maximum a posteriori probability, and Bayesian least mean square estimation. 4. Bayesian inference and examples of parameter estimation for Poisson, Gaussian, and multinomial distributions with conjugate priors.

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    This document discusses probabilistic reasoning and includes the following topics: 1. Data processing including data types, training models using supervised and unsupervised learning, and periodic model retraining. 2. Bayesian concepts including Bayes' rule, total probability, and comparing Bayesian and frequentist statistics. 3. Parameter estimation methods including maximum likelihood estimation, maximum a posteriori probability, and Bayesian least mean square estimation. 4. Bayesian inference and examples of parameter estimation for Poisson, Gaussian, and multinomial distributions with conjugate priors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    26 views55 pages

    ECE 368 Review Part 1

    Uploaded by

    Jialun Lyu
    This document discusses probabilistic reasoning and includes the following topics: 1. Data processing including data types, training models using supervised and unsupervised learning, and periodic model retraining. 2. Bayesian concepts including Bayes' rule, total probability, and comparing Bayesian and frequentist statistics. 3. Parameter estimation methods including maximum likelihood estimation, maximum a posteriori probability, and Bayesian least mean square estimation. 4. Bayesian inference and examples of parameter estimation for Poisson, Gaussian, and multinomial distributions with conjugate priors.

    Copyright:

    © All Rights Reserved
    Download as PDF, TXT or read online from Scribd
    Download as pdf or txt
    of 55

    Probabilistic Reasoning:

    ECE 368 Probabilistic Reasoning


    Alberto Leon-Garcia
    Week 1

    Probabilistic Reasoning 1
    Data, Learning and Model Development

    • Data: Scalar, Vector, Temporal, Spatial, Composite


    • Training: Use data to develop models
    • Supervised: Labeled Dataset (input and output data)
    • Unsupervised (output data)
    • Periodic Retraining
    • Reinforcement (state, action, reward)

    Dataset Training Model

    Data Model Inference


    Bayes Rule
    • Product Formula
    • P[A|B], likelihood
    • A Priori P[B]
    • A Posteriori Probability P[B|A]
    Total Probability
    • Sum Formula
    • Total Probability for P[A]
    Bayesian Statistics and Frequentist Statistics

    𝑃 𝐷 𝐻𝑖 𝑃 𝐻𝑖
    𝑃 𝐻𝑖 𝐷 = ~𝑃 𝐷 𝐻𝑖 𝑃 𝐻𝑖
    𝑃𝐷

    Frequentist Bayesian
    • A priori probability not used • Requires A priori probability over
    • Hypotheses not usually result of an hypotheses
    experiment • Must know or construct
    • Objective assessment of evidence “subjective” prior
    • Confidence intervals & p-values • Can explore different priors
    • A posteriori probability not used • Computationally intensive
    • Less computationally intensive • Learns as data accumulated
    • A posteriori enables decisions
    Conditional PMF and pdf
    Variance
    Covariance and Correlation Coefficient
    Covariance and Correlation Coefficient
    Independent Random Variables
    Maximum Likelihood & Bayesian
    Parameter Estimation
    ECE 368 Probabilistic Reasoning
    Week 2
    Agenda

    • Sample Mean and Convergence Properties


    • Estimating parameters of a Random Variable
    • ML Estimation and Frequentist Interpretation
    • MAP Estimation and Bayesian Interpretation
    • Bayesian Least Mean Square (LMS) Estimation
    Sum of Independent Identically Distributed RVs
    Weak Law of Large Numbers
    Parameter Estimation

    • Assume IID sequence of RVs


    • Estimate a parameter of X

    • Example: Bernoulli RV
    Properties of Estimators

    • Estimation error and Bias


    Maximum Likelihood Estimation

    • Likelihood
    Maximum Likelihood Estimation

    • Log likelihood
    MLE Bernoulli RV
    Laplace: Will the Sun Rise Tomorrow?

    Frequentist Bayesian
    Bayesian Statistical Inference
    Bayes Inference
    • Prior Distribution on ⍬ • Conditional distribution given ⍬
    Maximum A Posteriori Probability Rule
    MAP Estimate for Binomial with Beta Prior
    MAP Rule for Prediction
    More on ML, MAP, LMS
    Estimators
    ECE 368 Probabilistic Reasoning
    Week 3
    Agenda

    • Comparison of ML, MAP, LMS & Conditional Expectation


    • Poisson RV with Gamma Prior
    • Gaussian RV with Gaussian Prior
    • Multinomial RV with Dirichlet Prior

    • Strongly Recommended Reading:


    • Bertsekas: Chapter 8 Bayesian Inference; Ch. 9 ML & Frequentist
    • Bishop: 1.2.1 to 1.2.4
    • Murphy 3.3
    LMS & Conditional Expectation
    Important Conjugate Priors
    ML Estimator for Poisson RV
    • The number of defects in a VLSI chip is a Poisson RV with rate r
    defects per chip. Find the ML estimator for r.
    MAP Estimator: Poisson RV with Gamma Prior
    • Suppose the rate has a Gamma prior. Find the MAP estimator.
    LMS Estimator: Poisson RV with Gamma Prior
    ML for Gaussian RV
    • Find ML estimators for the mean & variance of a Gaussian RV
    Sample Variance
    MAP for Gaussian RV with Gaussian Prior
    • Find the MAP estimator for a Guassian RV w Gaussian Prior
    MAP for Multinomial RV with Dirichlet Prior
    Summary:

    • We elaborated on ML, MAP, and LMS estimation


    • Poisson with Gamma prior
    • Gaussian with Gaussian prior
    • Multinomial with Dirichlet prior
    • Conjugate priors greatly simplify computations
    Bayesian Hypothesis Testing
    ECE 368 Probabilistic Reasoning
    Week 4
    Agenda:

    • Binary Hypothesis Testing


    • Likelihood Ratio Test & Neyman Pearson Lemma
    • Significance Testing
    • Bayesian Hypothesis Testing
    • MAP Rule
    • Minimum Cost Decisions
    • ROC Curves
    • Naïve Bayes’ Classifier
    • ML Estimator
    • Laplace Smoothing
    Binary Hypothesis Testing
    Likelihood Ratio Test
    Neyman Pearson Lemma
    Significance Testing
    Bayesian Hypothesis Testing
    Naïve Bayes Assumption
    Summary

    • Binary Hypothesis Testing


    • Likelihood Ratio Test & Neyman Pearson Lemma
    • Significance Testing
    • Bayesian Hypothesis Testing
    • MAP Rule
    • Minimum Cost Decisions
    • ROC Curves
    • Naïve Bayes’ Classifier
    • ML Estimator
    • Laplace Smoothing

    You might also like