Home page Blog page

usage deeplearning
2019/07/11

linux服务器端 + cuda10 + anaconda3 + py35 + opencv3 之 caffe安装

本文算是caffe踩坑记录，折腾了两天，最终解决了，欣喜之余赶紧记录一波。

前提条件：

  • 已安装cuda , 但cudnn没有安装（因为没权限）
    • 更正，使用conda install cudnn即可安装，编译caffe的时候打开cudnn加速也是可以的
  • 无法使用sudo权限
  • 没有图形界面，是服务器端

0. 如何查看版本号

  1. 查看cuda版本 
    1
    cat /usr/local/cuda/version.txt
  2. 查看cudnn版本 
    1
    cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2
  3. 查看python版本 
    1
    python -V
  4. 查看opencv版本 
    1
    2
    3
    进入python
    import cv2
    cv2.__version__

1. 安装anaconda

上官网 https://www.anaconda.com/distribution/, 选择对应的anaconda版本，这里我选择的是linux + python 3.7 版本，使用wget进行下载：

1
wget https://repo.anaconda.com/archive/Anaconda3-2019.03-Linux-x86_64.sh

然后进行安装 sh Anaconda3-2019.03-Linux-x86_64.sh，注意记得修改安装路径，一般不建议装到/里，因为linux系统挂载方式决定了/相当于是win的C盘，最好是放在/data*/里面。

完成后修改用户环境变量vim ~/.bashrc，在文件末尾添加anaconda3的安装路径，例如：

1
2
# add anaconda env
export PATH=/data*/***/tool/anaconda3/bin:$PATH

接着更新环境变量source ~/.bashrc

2. 安装编译环境

  1. 新建虚拟环境
    如果不想多个项目依赖冲突，还是建立个虚拟环境把各个项目给隔开吧 
    1
    conda create -n caffe_py35 python=3.5
    这里-n代表命名，python=3.5规定python的版本号
  • conda env list 输出所有虚拟环境名
  • conda activate XXX 激活XXX环境
  • conda deactivate 关闭已激活的XXX环境
  • conda remove -n XXX --all 删除XXX环境

因此此处键入conda activate caffe_py35，激活python3.5环境，这是终端应该会显示(caffe_py35)xxx@xxx: xxx$，查看python的版本号也可以发现，是py35版。

  1. 安装opencv
1
conda install opencv

这里py35对应的是opencv3，所以也不需要刻意规定opencv的版本，可通过查看opencv版本的方法判断是否成功安装。

3. 编译caffe

  1. 进入caffe_py35虚拟环境

  2. 找个好地方把caffe源码git下来，不知道是下在release版，还是直接git好，这里我就直接git了

    1
    git https://github.com/BVLC/caffe.git

  3. 进入caffe目录，配置Makefile.config

巨坑预警

这里把自带的config复制过来

1
cp Makefile.config.example Makefile.config

在它的基础上进行修改。这时候你得仔细阅读其中的每一项，找到有关python和anaconda的行，进行修改操作，修改总结一下，大致有6处：

1
2
3
4
5
# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3
# ---------------------
# 因为是py35，因此这里需要修改
# ---------------------
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61
#-gencode arch=compute_20,code=sm_20 \
#-gencode arch=compute_20,code=sm_21 \
# ---------------------
# 因为是cuda10，所以把 *_20 and *_21 lines 删去
# ---------------------
1
2
3
4
5
6
7
8
9
10
11
12
13
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := ../anaconda3/envs/caffe_py35
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		$(ANACONDA_HOME)/include/python3.5m \
		$(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include
# ---------------------
# 这里是anaconda的安装位置，因为是虚拟环境，一定要注意ANACONDA_HOME的位置
# ---------------------
1
2
3
4
5
6
# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
PYTHON_LIBRARIES := boost_python-py35
# ---------------------
# 需要libboost，去/usr/lib/x86_64-linux-gnu/找一找libboost_python-py35.so.1.58.0是否匹配
# ---------------------
1
2
3
4
5
6
7
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial 
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
LINKFLAGS := -Wl,-rpath,$(PYTHON_LIB)
# ---------------------
# LINKFLAGS是 * [.build_release/tools/compute_image_mean.bin] Error 1 的坑，需要重新定义lib的位置
# ---------------------

这里我把我的Makefile.config贴出来供大家参考：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
# USE_CUDNN := 1
# ---------------------
# 需要用cudnn加速不，我这里没有cudnn，就没有取消注释
# ---------------------

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# uncomment to disable IO dependencies and corresponding data layers
# USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# This code is taken from https://github.com/sh1r0/caffe-android-lib
# USE_HDF5 := 0

# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
#	You should not set this flag if you will be reading LMDBs with any
#	possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1

# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3
# ---------------------
# 因为是py35，因此这里需要修改
# ---------------------

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# ---------------------
# cuda的安装位置，需要自行注意
# ---------------------

# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 through *_61 lines for compatibility.
# For CUDA < 8.0, comment the *_60 and *_61 lines for compatibility.
# For CUDA >= 9.0, comment the *_20 and *_21 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
		-gencode arch=compute_60,code=sm_60 \
		-gencode arch=compute_61,code=sm_61 \
		-gencode arch=compute_61,code=compute_61
#-gencode arch=compute_20,code=sm_20 \
#-gencode arch=compute_20,code=sm_21 \
# ---------------------
# 因为是cuda10，所以把 *_20 and *_21 lines 删去
# ---------------------

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
#		/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
ANACONDA_HOME := ../anaconda3/envs/caffe_py35
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		$(ANACONDA_HOME)/include/python3.5m \
		$(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include
# ---------------------
# 这里是anaconda的安装位置，因为是虚拟环境，一定要注意ANACONDA_HOME的位置
# ---------------------

# Uncomment to use Python 3 (default is Python 2)
# PYTHON_LIBRARIES := boost_python3 python3.5m
PYTHON_LIBRARIES := boost_python-py35
# ---------------------
# 需要libboost，去/usr/lib/x86_64-linux-gnu/找一找libboost_python-py35.so.1.58.0是否匹配
# ---------------------

# PYTHON_INCLUDE := /usr/include/python3.5m \
#                 /usr/lib/python3.5/dist-packages/numpy/core/include

# We need to be able to find libpythonX.X.so or .dylib.
# PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib
# ---------------------
# 这里是anaconda的安装位置
# ---------------------

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial 
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
LINKFLAGS := -Wl,-rpath,$(PYTHON_LIB)
# ---------------------
# LINKFLAGS是 * [.build_release/tools/compute_image_mean.bin] Error 1 的坑，需要重新定义lib的位置
# ---------------------

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# NCCL acceleration switch (uncomment to build with NCCL)
# https://github.com/NVIDIA/nccl (last tested version: v1.2.3-1+cuda8.0)
# USE_NCCL := 1

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1

# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

  1. make
  • make clean
  • make all
  • make test
  • make runtest
  • make pycaffe

看到一切都是 - RUN OK - ，舒服了

  1. 添加caffe到环境变量

vim ~/.bashrc，在末尾加入

1
export PYTHONPATH=/data*/***/tool/caffe/python

  1. import caffe的坑
    1. ImportError: No module named google.protobuf.internal
      到anaconda的bin文件夹下使用pip install protobuf，千万不要直接conda install，不然会没用的。
    2. ImportError: No module named ‘skimage’
      同样地，到anaconda的bin里面使用pip install scikit-image
    3. ImportError: libhdf5.so.101: cannot open shared object file: No such file or directory
      到环境依赖~/.bashrc里面进行修改，把在Makefile.config里面的libhdf5路径复制进去：
      1
      export LD_LIBRARY_PATH="/data*/***/tool/anaconda3/lib":$LD_LIBRARY_PATH
      记得source ~/.bashrc下。

4. 参考资料与致谢

  1. http://yingshu.ink/2017/01/12/Python3-5-Anaconda3-Caffe%E6%B7%B1%E5%BA%A6%E5%AD%A6%E4%B9%A0%E6%A1%86%E6%9E%B6%E6%90%AD%E5%BB%BA/
  2. https://blog.csdn.net/sinat_35406909/article/details/84198140
  3. https://blog.csdn.net/yhaolpz/article/details/71375762
  4. https://www.jianshu.com/p/1e405b9fe973
  5. https://blog.csdn.net/u011534057/article/details/51659999
Archive Tag About me
Total : 23
2020
2019
2018
2017
2016
diary wp intro kaggle paper cite algorithm mapreduce python cpp func review TOG SIGIR recommendation ICDE usage ubuntu TIP deeplearning
Total : 3